2023-11-27 13:33:31.285 INFO 10 --- [main,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2023-11-27 13:33:31.291 INFO 10 --- [main,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2023-11-27 13:33:31.291 INFO 10 --- [main,,] c.o.o.b.spring.BootstrapRunListener : set system property ocp.version to 4.0.3 2023-11-27 13:33:31.297 INFO 10 --- [main,,] c.o.o.b.spring.BootstrapRunListener : set systemProperties as the first PropertySource 2023-11-27 13:33:31.297 INFO 10 --- [main,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.0.3 2023-11-27 13:33:31.298 INFO 10 --- [main,,] com.oceanbase.ocp.OcpServerApplication : The following 2 profiles are active: "composite", "default" 2023-11-27 13:33:31.352 INFO 10 --- [main,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2023-11-27 13:33:40.403 INFO 10 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode 2023-11-27 13:33:40.404 INFO 10 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. 2023-11-27 13:33:40.725 INFO 10 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 281 ms. Found 3 JPA repository interfaces. 2023-11-27 13:33:40.728 INFO 10 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode 2023-11-27 13:33:40.728 INFO 10 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. 2023-11-27 13:33:42.030 INFO 10 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 1302 ms. Found 114 JPA repository interfaces. 2023-11-27 13:33:42.037 INFO 10 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode 2023-11-27 13:33:42.037 INFO 10 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. 2023-11-27 13:33:42.353 INFO 10 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 315 ms. Found 18 JPA repository interfaces. 2023-11-27 13:33:46.705 INFO 10 --- [main,,] o.s.cloud.context.scope.GenericScope : BeanFactory id=c94aeeaa-2c16-3a1f-a704-818f44fff60c 2023-11-27 13:33:48.385 INFO 10 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'localSecurityConfiguration.MethodSecurityConfiguration' of type [com.oceanbase.ocp.core.security.conf.LocalSecurityConfiguration$MethodSecurityConfiguration$$EnhancerBySpringCGLIB$$ea9197c6] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2023-11-27 13:33:48.443 INFO 10 --- [main,,] c.o.o.c.s.c.LocalSecurityConfiguration : Method-level access control enabled. 2023-11-27 13:33:50.164 INFO 10 --- [main,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] driver class=com.oceanbase.jdbc.Driver 2023-11-27 13:33:50.164 INFO 10 --- [main,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] username=meta_user@meta_tenant#obcluster 2023-11-27 13:33:50.164 INFO 10 --- [main,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] jdbc url=jdbc:oceanbase://133.197.206.97:2883/meta_database?useUnicode=true&characterEncoding=UTF8&connectTimeout=2000&socketTimeout=30000&rewriteBatchedStatements=true 2023-11-27 13:33:50.164 INFO 10 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : begin AfterDataSourceCreationHook 2023-11-27 13:33:50.164 INFO 10 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : end AfterDataSourceCreationHook 2023-11-27 13:33:51.024 INFO 10 --- [main,,] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default] 2023-11-27 13:33:51.570 INFO 10 --- [main,,] org.hibernate.Version : HHH000412: Hibernate ORM core version 5.6.12.Final 2023-11-27 13:33:52.994 INFO 10 --- [main,,] o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {5.1.2.Final} 2023-11-27 13:33:56.705 INFO 10 --- [main,,] com.alibaba.druid.pool.DruidDataSource : {dataSource-2,metadb-connect-pool} inited 2023-11-27 13:33:56.844 INFO 10 --- [main,,] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.MySQL5Dialect 2023-11-27 13:33:58.409 WARN 10 --- [main,,] o.h.t.d.java.JavaTypeDescriptorRegistry : HHH000481: Encountered Java type [class com.fasterxml.jackson.databind.JsonNode] for which we could not locate a JavaTypeDescriptor and which does not appear to implement equals and/or hashCode. This can lead to significant performance problems when performing equality/dirty checking involving this Java type. Consider registering a custom JavaTypeDescriptor or at least implementing equals/hashCode. 2023-11-27 13:33:59.725 WARN 10 --- [main,,] org.hibernate.mapping.RootClass : HHH000038: Composite-id class does not override equals(): com.oceanbase.ocp.core.metadb.repo.iam.jdbcsession.SpringJdbcSessionAttributeKey 2023-11-27 13:33:59.725 WARN 10 --- [main,,] org.hibernate.mapping.RootClass : HHH000039: Composite-id class does not override hashCode(): com.oceanbase.ocp.core.metadb.repo.iam.jdbcsession.SpringJdbcSessionAttributeKey 2023-11-27 13:34:04.334 INFO 10 --- [main,,] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform] 2023-11-27 13:34:04.408 INFO 10 --- [main,,] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default' 2023-11-27 13:34:10.332 INFO 10 --- [main,,] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http) 2023-11-27 13:34:10.371 INFO 10 --- [main,,] o.apache.catalina.core.StandardService : Starting service [Tomcat] 2023-11-27 13:34:10.372 INFO 10 --- [main,,] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.68] 2023-11-27 13:34:10.926 INFO 10 --- [main,,] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext 2023-11-27 13:34:10.927 INFO 10 --- [main,,] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 39562 ms 2023-11-27 13:34:12.080 INFO 10 --- [main,,] c.o.o.c.distributed.server.LocalServer : Local server, id=10.120.0.1:8080, name=68520c4efb 2023-11-27 13:34:13.483 INFO 10 --- [main,,] c.o.ocp.ha.maz.OcpMultiAzoneServiceImpl : OCP multi-azone not enabled 2023-11-27 13:34:13.500 INFO 10 --- [main,,] c.o.o.s.c.w.MonitordbDataSourceWrapper : [monitordb] driver class=com.oceanbase.jdbc.Driver 2023-11-27 13:34:13.500 INFO 10 --- [main,,] c.o.o.s.c.w.MonitordbDataSourceWrapper : [monitordb] username=monitor_user@monitor_tenant#obcluster 2023-11-27 13:34:13.500 INFO 10 --- [main,,] c.o.o.s.c.w.MonitordbDataSourceWrapper : [monitordb] jdbc url=jdbc:oceanbase://133.197.206.97:2883/monitor_database?useUnicode=true&characterEncoding=utf-8&connectTimeout=2000&socketTimeout=120000&rewriteBatchedStatements=true&allowMultiQueries=true 2023-11-27 13:34:13.545 INFO 10 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : begin AfterDataSourceCreationHook 2023-11-27 13:34:13.545 INFO 10 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : end AfterDataSourceCreationHook 2023-11-27 13:34:13.622 INFO 10 --- [main,,] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default] 2023-11-27 13:34:16.316 INFO 10 --- [main,,] com.alibaba.druid.pool.DruidDataSource : {dataSource-3,monitordb-connect-pool} inited 2023-11-27 13:34:16.317 INFO 10 --- [main,,] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.MySQL5Dialect 2023-11-27 13:34:16.975 INFO 10 --- [main,,] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform] 2023-11-27 13:34:16.976 INFO 10 --- [main,,] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default' 2023-11-27 13:34:17.534 INFO 10 --- [main,,] c.o.o.s.initializer.DatabaseInitializer : Start migration, location=migration/meta 2023-11-27 13:34:17.927 INFO 10 --- [main,,] c.o.o.s.initializer.DatabaseInitializer : Finish migration, location=migration/meta 2023-11-27 13:34:17.927 INFO 10 --- [main,,] c.o.o.s.initializer.DatabaseInitializer : Start migration, location=migration/monitor 2023-11-27 13:34:17.928 INFO 10 --- [main,,] c.oceanbase.ocp.core.util.ResourceUtils : Cannot find any file in location: migration/monitor 2023-11-27 13:34:17.928 INFO 10 --- [main,,] c.o.o.s.initializer.DatabaseInitializer : Migration schemas is empty. 2023-11-27 13:34:35.991 INFO 10 --- [main,,] org.reflections.Reflections : Reflections took 8291 ms to scan 67 urls, producing 858 keys and 6230 values 2023-11-27 13:34:44.113 INFO 10 --- [main,,] org.reflections.Reflections : Reflections took 8041 ms to scan 67 urls, producing 78 keys and 6163 values 2023-11-27 13:34:44.168 INFO 10 --- [main,,] c.o.o.server.analyzer.OcpStartUpChecker : audit event is disable 2023-11-27 13:34:44.720 INFO 10 --- [main,,] c.o.o.s.o.operator.LocalFileOperator : FileStorageService init, fileDirPath=/home/admin/data/files 2023-11-27 13:34:48.057 INFO 10 --- [main,,] c.o.ocp.server.config.BeanConfiguration : begin set auto discover false. 2023-11-27 13:34:48.062 INFO 10 --- [main,,] c.o.ocp.server.config.BeanConfiguration : set auto discover false success. 2023-11-27 13:34:51.399 INFO 10 --- [main,,] com.oceanbase.ocp.core.util.BeanUtils : BeanUtils initialized 2023-11-27 13:34:53.494 INFO 10 --- [main,,] c.o.o.c.s.service.LoginKeyService : Validate and set keys, publicKey=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCJcjN0iTXVhV07zUpEIco6zhJkLWM8FYOtCJceKjMUh+Qtji+u/t2EmeG+PU1zF1zV2tyHRHj7WhnUSWfdiRKiMuBhg4tz7PiIPgliMkp/V8dWR4viivqny5ZzxV+rwY/FXuqPOGIutKWADeXmye4fdmkHdxxCLsSPAdgKuZAzKwIDAQAB, valid=true 2023-11-27 13:34:53.494 INFO 10 --- [main,,] c.o.o.c.s.service.LoginKeyService : Login key is legal. 2023-11-27 13:34:54.181 INFO 10 --- [main,,] c.o.o.s.a.c.AlarmConfigurationManager : defaultTimeZone not set, use system default, zoneId=Asia/Shanghai 2023-11-27 13:34:54.594 INFO 10 --- [main,,] c.o.o.monitor.MonitorBeanConfiguration : ocp-monitor bean configuration done 2023-11-27 13:34:54.759 INFO 10 --- [main,,] c.o.o.monitor.MonitorBeanConfiguration : Table switch time is 1672704726 2023-11-27 13:34:55.919 INFO 10 --- [main,,] c.o.o.s.alarm.AlarmBeanConfiguration : scheduleEnabled=true 2023-11-27 13:34:56.260 INFO 10 --- [main,,] c.o.o.s.a.s.OcpAlarmInitializeService : initialize global tasks done 2023-11-27 13:34:56.260 INFO 10 --- [main,,] c.o.o.s.a.s.OcpAlarmInitializeService : initialize done 2023-11-27 13:34:57.838 INFO 10 --- [main,,] c.o.o.s.a.core.silence.GlobalSilencer : enabled alarm types changed, previous=, current=ob_host_task_timeout,ob_tenant_long_trans_exist,ob_host_net_send_percent_over_threshold,host_agent_goroutine_count_over_threshold,tenant_cpu_percent_over_threshold,monitor_exporter_unavaliable,ob_tenant_compaction_error,odp_instance_expanse_failed,ob_cluster_exists_inactive_server,ob_host_load1_per_cpu_over_threshold,ob_cluster_frozen_version_delta_over_threshold,no_enough_exporter,inc_backup_delay,host_ntp_service_not_exist,message_log_alarm,host_disk_readonly,system_parameter_not_taken_effect,node_file_root_usage,rootservice_log_alarm,same_alarm_rule_detect_too_many_targets,os_tsar_traffic_overload,ob_cluster_sync_delay_time_too_long,ob_cluster_merge_timeout,partition_create_failed,node_file_inode_usage,obproxy_process_dead,os_tsar_cpu_sys_abnormal,ob_cluster_exists_index_fail_table,ob_tenant_operation_info,base_backup_timeout,host_ntp_offset_too_large,os_nic_1000m_hwm,ob_cluster_no_frozen,vpc_connect_failed,os_cpu_irq_error,ob_cluster_status_check_failed,host_agent_res_memory_over_threshold,host_agent_open_fd_count_over_threshold,ob_cluster_no_merge,ob_cluster_sync_failed,ocp_http_request_timeout,odp_instance_compress_failed,backup_process_dead,ob_host_log_disk_percent_over_threshold,ob_host_install_disk_percent_over_threshold,ic_server_connect_failed,os_nic_1000m_full,ob_tenant500_mem_hold_percent_over_threshold,odp_sql_execute_failed,obagent_dead,ob_server_sstable_percent_over_threshold,backup_storage_capacity_over_threshold,os_tsar_traffic_drop,tenant_active_memstore_percent_over_threshold,ob_host_data_disk_percent_over_threshold,os_tsar_cpu_util_full,ob_tenant_exists_expired_xa_trans,os_tsar_cpu_util_hwm,os_tsar_traffic_error,ob_cluster_operation_info,ob_cluster_merge_error,backup_storage_capacity_timeout_or_interrupted,ob_tenant_large_trans_exist,host_unavailable,ob_host_mem_percent_over_threshold,os_observer_not_exist,ocp_remote_server_time_diff_too_large,obproxy_log_alarm,ob_tenant_expired_trans_exist,os_kernel_io_hang,ob_tenant_task_timeout,os_tsar_sda_ioawait,odp_sql_query_slow,os_tsar_nvme_ioawait,tenant_memstore_percent_over_threshold,election_log_alarm,ob_zone_sstable_percent_over_threshold,obproxy_cannot_connected,observer_all_error_log_alarm,ocp_alarm_detect_timeout,ob_cluster_rs_not_same,ob_server_stopped,base_secondary_backup_fail,ob_host_cpu_percent_over_threshold,ob_cpu_percent_over_threshold,ocp_http_request_too_many_errors_occur,base_backup_too_long_time_no_one_success_task,oms_migrate_delay_over_threshold,ocp_meta_db_disconnected,ob_host_disk_readonly,ob_host_net_recv_percent_over_threshold,ob_host_connection_percent_over_threshold,ob_tenant_no_compaction,backup_storage_capacity_retry_times_exceeded,obproxyd_process_dead,ob_tenant_log_disk_usage_high,ob_tenant_no_frozen,observer_log_alarm,ob_tenant500_mem_hold_over_threshold,os_observer_fd_usage,ob_cannot_connected,ob_host_tcp_retrans_percent_over_threshold,base_backup_fail,os_observer_core_dump,ob_host_partition_count_over_threshold 2023-11-27 13:35:02.646 INFO 10 --- [main,,] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:35:03.316 INFO 10 --- [main,,] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:35:03.709 INFO 10 --- [main,,] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:35:04.079 INFO 10 --- [main,,] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:35:04.328 INFO 10 --- [main,,] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:35:04.540 INFO 10 --- [main,,] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:35:04.746 INFO 10 --- [main,,] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:35:04.953 INFO 10 --- [main,,] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:35:05.204 INFO 10 --- [main,,] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:35:05.423 INFO 10 --- [main,,] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:35:05.616 INFO 10 --- [main,,] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:35:05.801 INFO 10 --- [main,,] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:35:05.990 INFO 10 --- [main,,] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:35:06.181 INFO 10 --- [main,,] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:35:06.393 INFO 10 --- [main,,] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:35:06.584 INFO 10 --- [main,,] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:35:06.739 INFO 10 --- [main,,] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 781 2023-11-27 13:35:06.931 INFO 10 --- [main,,] c.o.o.s.a.service.OcpAlarmSearchService : total load alarm history size: 16781 2023-11-27 13:35:06.931 INFO 10 --- [main,,] c.o.o.s.a.service.OcpAlarmSearchService : refresh alarm search done 2023-11-27 13:35:07.000 INFO 10 --- [main,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updater scheduler started on 10.120.0.1:8080 2023-11-27 13:35:07.005 INFO 10 --- [main,,] c.o.o.s.o.OdpClientOperationService : Init local odp rich client proxyro password 2023-11-27 13:35:07.882 INFO 10 --- [main,,] c.o.o.s.o.OdpClientOperationService : Update cluster:darenOB proxyro password 2023-11-27 13:35:09.993 INFO 10 --- [main,,] c.o.o.c.odpclient.JdbcOdpClientOperator : Begin to update all slots proxyro password, clusterName:darenOB 2023-11-27 13:35:09.994 INFO 10 --- [main,,] c.o.o.c.odpclient.JdbcOdpClientOperator : Update password slot observer_sys_password, clusterName:darenOB 2023-11-27 13:35:10.003 INFO 10 --- [main,,] c.o.o.c.odpclient.JdbcOdpClientOperator : Update password slot observer_sys_password1, clusterName:darenOB 2023-11-27 13:35:10.009 INFO 10 --- [main,,] c.o.o.s.o.OdpClientOperationService : Success to update proxyro password of cluster:darenOB 2023-11-27 13:35:12.004 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:35:12.340 INFO 10 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_config_template.yaml in jar 2023-11-27 13:35:12.476 INFO 10 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_ob_relations_config.yaml in jar 2023-11-27 13:35:12.519 INFO 10 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_ability_ob_version_relation_config.yaml in jar 2023-11-27 13:35:12.542 INFO 10 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from physical_backup_ability_ob_version_relation_config.yaml in jar 2023-11-27 13:35:12.532 WARN 10 --- [ocp-updater-0,,] c.o.o.c.d.server.DefaultServerUpdater : Deleted expired server: DistributedServer{id=10.120.0.1:8080, name=68520c4efb, expireInSeconds=30, createTime=2023-11-20T07:41:06+08:00, updateTime=2023-11-24T10:49:26+08:00, expireTime=2023-11-24T10:49:56+08:00, isActive=false} 2023-11-27 13:35:16.299 INFO 10 --- [main,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Init eventTemplateCache: 172 cache entries, 172 elements. 2023-11-27 13:35:17.915 INFO 10 --- [main,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService updater scheduler started on 10.120.0.1:8080 2023-11-27 13:35:17.930 INFO 10 --- [main,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : Init, apiConnectTimeoutMillis:1000, apiReadTimeoutMillis:1000, taskSchedulePeriodSec:3 2023-11-27 13:35:18.256 WARN 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : Error state, the follower found itself leader. 2023-11-27 13:35:18.256 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :null 2023-11-27 13:35:20.857 INFO 10 --- [pool-heartbeat-metric-collect-10.120.0.1:8080-11,,] c.o.o.c.s.registry.ConsumerRegistry : Slice modified, module=metric-collect, name=10.120.0.1:8080-1, consumerId=1, resources=[2000111, 2000112, 2000113, 2000117, 2000118, 2000124, 2000125, 2000126, 2000137, 2000143, 2000144, 2000145, 2000149, 2000150, 2000151, 2000152, 2000157, 2000158, 2000182, 2000183, 2000184, 2000188, 2000189, 2000190] 2023-11-27 13:35:20.858 INFO 10 --- [pool-heartbeat-metric-collect-10.120.0.1:8080-01,,] c.o.o.c.s.registry.ConsumerRegistry : Slice modified, module=metric-collect, name=10.120.0.1:8080-0, consumerId=2, resources=[2000110, 2000114, 2000115, 2000116, 2000122, 2000123, 2000127, 2000128, 2000129, 2000133, 2000134, 2000135, 2000136, 2000141, 2000142, 2000153, 2000159, 2000160, 2000161, 2000180, 2000181, 2000185, 2000186, 2000187, 2000191, 2000192, 2000193, 2000194] 2023-11-27 13:35:21.266 WARN 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : Error state, the follower found itself leader. 2023-11-27 13:35:21.266 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :null 2023-11-27 13:35:21.290 INFO 10 --- [main,,] c.o.o.s.a.c.n.NotificationSender : NotificationSender init, scriptDirectoryPath=/home/admin/ocp-server/alarm-channel-script 2023-11-27 13:35:21.496 INFO 10 --- [main,,] c.o.o.c.t.engine.util.ThreadPrintStream : replaceSystemOut done 2023-11-27 13:35:21.580 INFO 10 --- [main,,] c.o.o.s.a.c.AlarmConfiguration : alarm configuration set from system properties 2023-11-27 13:35:22.658 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:35:22.916 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.120.0.1:8080 2023-11-27 13:35:22.952 WARN 10 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=CMDB_SYNCER, serverId=10.120.0.1:8080, expireInSeconds=30, createTime=2023-11-20T07:41:16+08:00, updateTime=2023-11-24T10:49:33+08:00, expireTime=null, isLeader=false} 2023-11-27 13:35:22.976 WARN 10 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=MC_MANAGER, serverId=10.120.0.1:8080, expireInSeconds=30, createTime=2023-11-20T07:41:16+08:00, updateTime=2023-11-24T10:49:33+08:00, expireTime=null, isLeader=false} 2023-11-27 13:35:22.995 WARN 10 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=PERF_SQL_ROLLUP, serverId=10.120.0.1:8080, expireInSeconds=30, createTime=2023-11-20T07:41:16+08:00, updateTime=2023-11-24T10:49:21+08:00, expireTime=null, isLeader=false} 2023-11-27 13:35:22.997 WARN 10 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=PLAN_CHANGE_DETECTOR, serverId=10.120.0.1:8080, expireInSeconds=30, createTime=2023-11-24T10:45:07+08:00, updateTime=2023-11-24T10:49:07+08:00, expireTime=null, isLeader=false} 2023-11-27 13:35:22.999 WARN 10 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=RATE_LIMIT, serverId=10.120.0.1:8080, expireInSeconds=30, createTime=2023-11-20T07:41:16+08:00, updateTime=2023-11-24T10:49:33+08:00, expireTime=null, isLeader=false} 2023-11-27 13:35:23.002 WARN 10 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=SQL_INDEX_DIAGNOSER, serverId=10.120.0.1:8080, expireInSeconds=30, createTime=2023-11-24T10:37:36+08:00, updateTime=2023-11-24T10:49:07+08:00, expireTime=null, isLeader=false} 2023-11-27 13:35:23.006 WARN 10 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=SQL_PERF_DIAGNOSER, serverId=10.120.0.1:8080, expireInSeconds=30, createTime=2023-11-24T10:44:37+08:00, updateTime=2023-11-24T10:49:07+08:00, expireTime=null, isLeader=false} 2023-11-27 13:35:23.007 WARN 10 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=TASK_MANAGER, serverId=10.120.0.1:8080, expireInSeconds=30, createTime=2023-11-20T07:41:16+08:00, updateTime=2023-11-24T10:49:33+08:00, expireTime=null, isLeader=false} 2023-11-27 13:35:23.011 WARN 10 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service SQL_INDEX_DIAGNOSER, Incorrect result size: expected 1, actual 0 2023-11-27 13:35:23.068 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:35:23.076 WARN 10 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service MC_MANAGER, Incorrect result size: expected 1, actual 0 2023-11-27 13:35:23.088 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.120.0.1:8080 2023-11-27 13:35:23.095 WARN 10 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service PERF_SQL_ROLLUP, Incorrect result size: expected 1, actual 0 2023-11-27 13:35:23.098 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:35:23.103 WARN 10 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service CMDB_SYNCER, Incorrect result size: expected 1, actual 0 2023-11-27 13:35:23.110 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:35:23.117 WARN 10 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service TASK_MANAGER, Incorrect result size: expected 1, actual 0 2023-11-27 13:35:23.126 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:35:23.130 WARN 10 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service RATE_LIMIT, Incorrect result size: expected 1, actual 0 2023-11-27 13:35:23.133 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.120.0.1:8080 2023-11-27 13:35:23.136 WARN 10 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service PLAN_CHANGE_DETECTOR, Incorrect result size: expected 1, actual 0 2023-11-27 13:35:23.140 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.120.0.1:8080 2023-11-27 13:35:23.143 WARN 10 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service SQL_PERF_DIAGNOSER, Incorrect result size: expected 1, actual 0 2023-11-27 13:35:24.269 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 gets the leadership... 2023-11-27 13:35:25.314 INFO 10 --- [main,,] c.o.o.m.c.MonitorDataRolloverProperties : Monitor retention conf is {ob_cluster_system_event=31, ocp_metric_data_1=8, ocp_metric_data_60=31, metric_hour_data=1098, metric_daily_data=1098} 2023-11-27 13:35:25.369 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Metadb, table=profile_credential_sharing 2023-11-27 13:35:25.370 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Metadb, table=ocp_cache_info 2023-11-27 13:35:25.370 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat_0 2023-11-27 13:35:25.370 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat_1 2023-11-27 13:35:25.370 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat_2 2023-11-27 13:35:25.370 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat 2023-11-27 13:35:25.370 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sqltext 2023-11-27 13:35:25.370 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_high_risk_sql 2023-11-27 13:35:25.370 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0 2023-11-27 13:35:25.370 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_plan_cache_plan 2023-11-27 13:35:25.370 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_plan_cache_plan_explain 2023-11-27 13:35:25.370 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_sample 2023-11-27 13:35:25.370 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics 2023-11-27 13:35:25.370 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_perf_sql_diagnosis_task 2023-11-27 13:35:25.371 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_perf_sql_diagnosis 2023-11-27 13:35:25.371 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_active_session_stat_0 2023-11-27 13:35:25.371 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_snapshot 2023-11-27 13:35:25.371 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_cluster 2023-11-27 13:35:25.371 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_tenant 2023-11-27 13:35:25.371 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_server 2023-11-27 13:35:25.371 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_replica 2023-11-27 13:35:25.371 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sysstat 2023-11-27 13:35:25.371 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_row_lock 2023-11-27 13:35:25.371 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_latch 2023-11-27 13:35:25.371 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_cache 2023-11-27 13:35:25.371 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_parameter 2023-11-27 13:35:25.371 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_rpc 2023-11-27 13:35:25.371 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_packet 2023-11-27 13:35:25.371 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_clog 2023-11-27 13:35:25.372 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_location_cache 2023-11-27 13:35:25.372 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_gts 2023-11-27 13:35:25.372 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_system_event 2023-11-27 13:35:25.372 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_memory 2023-11-27 13:35:25.372 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_partition_hit 2023-11-27 13:35:25.372 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_plan_monitor 2023-11-27 13:35:25.372 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_trans_stat 2023-11-27 13:35:25.372 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_cluster_system_event 2023-11-27 13:35:25.372 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_metric_data_1 2023-11-27 13:35:25.372 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_metric_data_60 2023-11-27 13:35:25.372 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_hour_data 2023-11-27 13:35:25.372 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_daily_data 2023-11-27 13:35:25.372 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_second 2023-11-27 13:35:25.373 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_minute 2023-11-27 13:35:25.373 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_hour 2023-11-27 13:35:25.373 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_daily 2023-11-27 13:35:25.373 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=sql_aggregate_data 2023-11-27 13:35:25.373 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_partition_stats_compressed 2023-11-27 13:35:25.373 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_tenant_mini_compaction_info 2023-11-27 13:35:25.384 INFO 10 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167004] Submitting initial worker 2023-11-27 13:35:25.458 INFO 10 --- [main,,] c.o.o.s.analyzer.SslTrustInitializer : ssl not enabled, do nothing 2023-11-27 13:35:25.705 INFO 10 --- [main,,] c.o.o.s.alarm.schedule.AlarmSchedules : AlarmSchedules construct done 2023-11-27 13:35:26.040 INFO 10 --- [main,,] c.o.o.s.a.c.d.metrics.AlarmMetricsQuery : createMetricMetaMapping done, metasCount=105 2023-11-27 13:35:26.366 INFO 10 --- [main,,] c.o.o.s.a.c.d.metrics.SqlMetricsQuery : createMetricMetaMapping done, metasCount=105 2023-11-27 13:35:26.433 INFO 10 --- [main,,] c.o.o.s.alarm.AlarmBeanConfiguration : initialize alarmDetectMetricExecutor 2023-11-27 13:35:26.510 INFO 10 --- [main,,] c.o.o.s.alarm.AlarmBeanConfiguration : initialize alarmTaskExecutor 2023-11-27 13:35:26.546 INFO 10 --- [main,,] c.o.o.s.a.utils.task.AlarmTaskManager : alarm task manager construct done, executorName=10.120.0.1:10 2023-11-27 13:35:26.741 INFO 10 --- [main,,] c.o.o.s.dataclean.TableCleaningService : Successfully init all table cleaning configurations: {"cleanTaskInstance":{"batchIntervalMillis":0,"batchSize":0,"condition":"create_time <= date_sub(curdate(), interval 30 day) and type = \"SYS_SCHEDULED\"","priority":2,"tableName":"task_instance"},"cleanSubTaskInstance":{"batchIntervalMillis":0,"batchSize":0,"condition":"task_id in (select id from task_instance where create_time <= date_sub(curdate(), interval 30 day) and type = \"SYS_SCHEDULED\")","priority":1,"tableName":"subtask_instance"},"cleanSubTaskLog":{"batchIntervalMillis":0,"batchSize":0,"condition":"subtask_id in (select id from subtask_instance where task_id in (select id from task_instance where create_time <= date_sub(curdate(), interval 30 day) and type = \"SYS_SCHEDULED\"))","priority":0,"tableName":"subtask_log"},"cleanSubTaskDependencies":{"batchIntervalMillis":0,"batchSize":0,"condition":"child_id in (select id from subtask_instance where task_id in (select id from task_instance where create_time <= date_sub(curdate(), interval 30 day) and type = \"SYS_SCHEDULED\"))","priority":0,"tableName":"subtask_dependencies"}} 2023-11-27 13:35:26.761 INFO 10 --- [main,,] c.o.o.s.alarm.AlarmBeanConfiguration : initialize notificationSendExecutor 2023-11-27 13:35:26.844 INFO 10 --- [main,,] c.o.o.s.o.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.print.sql = true 2023-11-27 13:35:26.845 INFO 10 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set sqlLogEnabled=true 2023-11-27 13:35:26.847 INFO 10 --- [main,,] c.o.o.s.o.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.max-idle.seconds = 3600 2023-11-27 13:35:26.887 INFO 10 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set maxIdleSeconds=3600 2023-11-27 13:35:26.894 INFO 10 --- [main,,] c.o.o.s.o.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.cleanup.period.seconds = 300 2023-11-27 13:35:26.895 INFO 10 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set cleanPeriod=300 2023-11-27 13:35:26.897 INFO 10 --- [main,,] c.o.o.s.o.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.operation.global.timeout.millis = 300000 2023-11-27 13:35:26.897 INFO 10 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set ddlGlobalTimeoutMillis=300000 2023-11-27 13:35:26.899 INFO 10 --- [main,,] c.o.o.s.o.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.init.timeout.millis = 3000 2023-11-27 13:35:26.900 INFO 10 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set initTimeoutMillis=3000 2023-11-27 13:35:26.901 INFO 10 --- [main,,] c.o.o.s.o.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.slow.query.threshold.millis = 1000 2023-11-27 13:35:26.901 INFO 10 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set slowQueryThresholdMillis=1000 2023-11-27 13:35:26.903 INFO 10 --- [main,,] c.o.o.s.o.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.holder.capacity = 100 2023-11-27 13:35:26.906 INFO 10 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set cacheSize=100 2023-11-27 13:35:26.908 INFO 10 --- [main,,] c.o.o.s.o.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.socket.read.timeout.millis = 1800000 2023-11-27 13:35:26.908 INFO 10 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set socketReadTimeoutMillis=1800000 2023-11-27 13:35:26.910 INFO 10 --- [main,,] c.o.o.s.o.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.init.executor.thread-count = 16 2023-11-27 13:35:26.910 INFO 10 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set socketInitThreadCount=16 2023-11-27 13:35:26.911 INFO 10 --- [main,,] c.o.o.s.o.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.socket.connect.timeout.millis = 2000 2023-11-27 13:35:26.912 INFO 10 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set socketConnectTimeoutMillis=2000 2023-11-27 13:35:27.269 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:35:27.389 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:35:27.635 WARN 10 --- [main,,] JpaBaseConfiguration$JpaWebConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning 2023-11-27 13:35:28.351 INFO 10 --- [main,,] f.a.AutowiredAnnotationBeanPostProcessor : Autowired annotation is not supported on static methods: public static void com.oceanbase.ocp.core.util.jpa.JsonNodeConverter.init(com.fasterxml.jackson.databind.ObjectMapper) 2023-11-27 13:35:28.840 INFO 10 --- [main,,] s.i.c.SqlAuditRawStatRollupDaemonService : SQL stat rollup daemon submitted 2023-11-27 13:35:28.840 INFO 10 --- [main,,] s.i.c.SqlAuditRawStatRollupDaemonService : SQL stat rollup workers submitted 2023-11-27 13:35:30.395 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:35:30.397 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:35:32.079 INFO 10 --- [main,,] c.o.o.s.s.b.BuiltInFilesInitializeLoader : Load lib files, localBuiltInDir=/home/admin/ocp-server/lib 2023-11-27 13:35:32.079 INFO 10 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Remove built-in file:pos.py 2023-11-27 13:35:32.234 INFO 10 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Delete object, bucket=built-in, name=pos.py, objectId=72 2023-11-27 13:35:32.336 INFO 10 --- [main,,] c.o.o.s.o.operator.ObjectBlockOperator : deleteByFileId, deletedBlockCount=1 2023-11-27 13:35:32.724 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:35:32.747 INFO 10 --- [main,,] c.o.o.s.o.operator.LocalFileOperator : delete local file, absolutePathName=/home/admin/data/files/built-in/pos.py 2023-11-27 13:35:32.747 INFO 10 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Load built-in file:pos.py 2023-11-27 13:35:33.089 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:35:33.097 INFO 10 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Init saving object, creatorId=200, bucket=built-in, name=pos.py, objectId=2000001 2023-11-27 13:35:33.111 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:35:33.126 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:35:33.134 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:35:33.188 INFO 10 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Finish saving object, objectId=2000001, name=pos.py 2023-11-27 13:35:33.188 INFO 10 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Remove built-in file:obproxyd.sh 2023-11-27 13:35:33.209 INFO 10 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Delete object, bucket=built-in, name=obproxyd.sh, objectId=73 2023-11-27 13:35:33.238 INFO 10 --- [main,,] c.o.o.s.o.operator.ObjectBlockOperator : deleteByFileId, deletedBlockCount=1 2023-11-27 13:35:33.256 INFO 10 --- [main,,] c.o.o.s.o.operator.LocalFileOperator : delete local file, absolutePathName=/home/admin/data/files/built-in/obproxyd.sh 2023-11-27 13:35:33.257 INFO 10 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Load built-in file:obproxyd.sh 2023-11-27 13:35:33.365 INFO 10 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Init saving object, creatorId=200, bucket=built-in, name=obproxyd.sh, objectId=2000002 2023-11-27 13:35:33.400 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:35:33.416 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:35:33.443 INFO 10 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Finish saving object, objectId=2000002, name=obproxyd.sh 2023-11-27 13:35:33.721 INFO 10 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:t-oceanbase-ocp-agent-4.0.3-20230412103036.alios7.aarch64.rpm 2023-11-27 13:35:33.817 INFO 10 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Software package already exist, name=t-oceanbase-ocp-agent-4.0.3-20230412103036.alios7.aarch64.rpm 2023-11-27 13:35:33.818 INFO 10 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:t-oceanbase-ocp-agent-4.0.3-20230412103152.alios7.x86_64.rpm 2023-11-27 13:35:33.829 INFO 10 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Software package already exist, name=t-oceanbase-ocp-agent-4.0.3-20230412103152.alios7.x86_64.rpm 2023-11-27 13:35:33.859 INFO 10 --- [main,,] c.o.ocp.ha.maz.OcpAzoneDaemonService : Multi azone daemon not enabled 2023-11-27 13:35:33.967 INFO 10 --- [main,,] c.o.ocp.monitor.OcpMonitorManager : OcpMonitorManager init start... 2023-11-27 13:35:33.969 INFO 10 --- [main,,] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1701063360, interval=60 2023-11-27 13:35:33.970 INFO 10 --- [main,,] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1701064800, interval=3600 2023-11-27 13:35:33.972 INFO 10 --- [main,,] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1701063360, interval=60 2023-11-27 13:35:33.973 INFO 10 --- [main,,] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1701064800, interval=3600 2023-11-27 13:35:33.975 INFO 10 --- [main,,] c.o.ocp.monitor.OcpMonitorManager : OcpMonitorManager init success. 2023-11-27 13:35:34.628 INFO 10 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_config_template.yaml in jar 2023-11-27 13:35:34.644 INFO 10 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_ob_relations_config.yaml in jar 2023-11-27 13:35:34.647 INFO 10 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_ability_ob_version_relation_config.yaml in jar 2023-11-27 13:35:34.647 INFO 10 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from physical_backup_ability_ob_version_relation_config.yaml in jar 2023-11-27 13:35:34.648 INFO 10 --- [main,,] c.o.o.b.i.c.BackupConfiguration : init backup agent meta cfg 2023-11-27 13:35:34.823 INFO 10 --- [main,,] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key global_lock_com.oceanbase.ocp.backup.internal.task.backupocptaskmanager 2023-11-27 13:35:34.830 INFO 10 --- [main,,] c.o.o.b.i.a.BackupRestoreAgentService : upgradeFromOldAgentToNewService start. 2023-11-27 13:35:34.979 INFO 10 --- [main,,] c.o.o.b.i.a.BackupRestoreAgentService : serviceInfos=[], allMachineList=[] 2023-11-27 13:35:35.043 INFO 10 --- [main,,] c.o.o.b.i.task.BackupOcpTaskManager : register backup schedules success 2023-11-27 13:35:35.472 INFO 10 --- [main,,] c.o.o.core.sharding.ShardingCoordinator : Sharding registry started, module=metric-collect 2023-11-27 13:35:35.945 INFO 10 --- [main,,] c.o.o.b.i.m.s.StorageExporterServiceImpl : collectEnabled=true,periodMinutes=180, register backup exporter. 2023-11-27 13:35:36.416 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:35:36.428 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:35:36.547 INFO 10 --- [main,,] c.o.o.s.o.s.c.OverviewBeanConfiguration : initialize overviewContainerExecutor 2023-11-27 13:35:36.569 INFO 10 --- [main,,] c.o.o.s.o.s.c.OverviewBeanConfiguration : initialize overviewServiceExecutor 2023-11-27 13:35:38.098 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.120.0.1:8080 2023-11-27 13:35:38.988 INFO 10 --- [main,,] o.s.b.a.w.s.WelcomePageHandlerMapping : Adding welcome page: class path resource [static/index.html] 2023-11-27 13:35:39.429 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:35:39.432 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:35:40.508 INFO 10 --- [part-initial-worker,,] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2023-11-27 13:35:40.510 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2023-11-27 13:35:40.529 INFO 10 --- [part-initial-worker,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=proxy, clusterName=obcluster, obClusterId=null, tenantName=monitor_tenant, username=monitor_user, address=133.197.206.97, port=2883, database=monitor_database) 2023-11-27 13:35:40.529 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=proxy, address=133.197.206.97, port=2883, obsAddrList=null, username=monitor_user, tenantName=monitor_tenant, clusterName=obcluster, obClusterId=null, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=monitor_database) 2023-11-27 13:35:40.843 INFO 10 --- [obsdk-init-thread-1,,128b41ad79f8] com.alibaba.druid.pool.DruidDataSource : {dataSource-4,ob-connector-datasource_proxy_obcluster_null_monitor_tenant_monitor_user_133.197.206.97_2883} inited 2023-11-27 13:35:40.843 INFO 10 --- [obsdk-init-thread-1,,128b41ad79f8] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource success, poolingCount=1, connectProperties=ConnectProperties(connectionMode=proxy, address=133.197.206.97, port=2883, obsAddrList=null, username=monitor_user, tenantName=monitor_tenant, clusterName=obcluster, obClusterId=null, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=monitor_database) 2023-11-27 13:35:40.843 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector success 2023-11-27 13:35:40.848 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:40.857 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:40.861 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:40.899 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2023-11-27 13:35:41.047 INFO 10 --- [pool-sharding-metric-collect1,,] c.o.o.c.s.registry.ShardingRegistry : Start rebalance, consumerSize=2, sliceEntitySize=2, resourceSize=52 2023-11-27 13:35:41.069 INFO 10 --- [pool-sharding-metric-collect1,,] c.o.o.c.s.registry.ShardingHelper : Mark consumer state, newState=PAUSE 2023-11-27 13:35:41.403 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:41.404 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:41.405 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:35:41.409 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:41.409 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:41.409 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2023-11-27 13:35:41.595 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_0, partition=P20231203 2023-11-27 13:35:41.603 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:41.603 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:41.610 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_0` add partition(partition P20231203 values less than (1701648000000000)) 2023-11-27 13:35:42.219 INFO 10 --- [pool-heartbeat-metric-collect-10.120.0.1:8080-01,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from PAUSE to WAITING. 2023-11-27 13:35:42.224 INFO 10 --- [pool-heartbeat-metric-collect-10.120.0.1:8080-11,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from PAUSE to WAITING. 2023-11-27 13:35:42.436 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:35:42.446 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:35:42.739 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:35:42.960 INFO 10 --- [pool-sharding-metric-collect1,,] c.o.o.c.s.registry.ShardingHelper : Mark consumer state, newState=ASSIGNED 2023-11-27 13:35:43.092 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:35:43.120 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:35:43.143 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:35:43.151 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:35:43.178 WARN 10 --- [main,,] ion$DefaultTemplateResolverConfiguration : Cannot find template location: classpath:/templates/ (please add some templates, check your Thymeleaf configuration, or set spring.thymeleaf.check-template-location=false) 2023-11-27 13:35:43.401 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] slow query, durationMillis=1791, sql=alter table `ob_hist_sql_audit_stat_0` add partition(partition P20231203 values less than (1701648000000000)) 2023-11-27 13:35:43.425 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:43.425 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:43.427 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_0' and table_type='BASE TABLE' 2023-11-27 13:35:44.004 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_0 2023-11-27 13:35:44.037 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_0, partition=P20231204 2023-11-27 13:35:44.045 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:44.045 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:44.061 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_0` add partition(partition P20231204 values less than (1701734400000000)) 2023-11-27 13:35:44.220 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:44.220 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:44.225 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_0' and table_type='BASE TABLE' 2023-11-27 13:35:44.286 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_0 2023-11-27 13:35:44.294 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_0, partition=P20231205 2023-11-27 13:35:44.303 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:44.303 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:44.308 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_0` add partition(partition P20231205 values less than (1701820800000000)) 2023-11-27 13:35:44.454 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:44.454 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:44.457 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_0' and table_type='BASE TABLE' 2023-11-27 13:35:44.506 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_0 2023-11-27 13:35:44.509 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_sql_audit_stat_0, partition=P20231122 2023-11-27 13:35:44.510 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:44.510 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:44.510 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_0` drop partition (P20231122) 2023-11-27 13:35:44.588 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:44.588 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:44.593 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_0' and table_type='BASE TABLE' 2023-11-27 13:35:44.600 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_sql_audit_stat_0, partition=P20231123 2023-11-27 13:35:44.602 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:44.602 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:44.603 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_0` drop partition (P20231123) 2023-11-27 13:35:44.660 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:44.660 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:44.661 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_0' and table_type='BASE TABLE' 2023-11-27 13:35:44.674 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_sql_audit_stat_0, partition=P20231124 2023-11-27 13:35:44.675 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:44.675 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:44.676 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_0` drop partition (P20231124) 2023-11-27 13:35:44.757 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:44.757 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:44.758 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_0' and table_type='BASE TABLE' 2023-11-27 13:35:44.769 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:44.770 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:44.773 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:35:44.775 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:44.775 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:44.776 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2023-11-27 13:35:44.813 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20231203 2023-11-27 13:35:44.814 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:44.815 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:44.815 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20231203 values less than (1701648000000000)) 2023-11-27 13:35:44.971 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:44.972 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:44.973 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2023-11-27 13:35:45.006 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2023-11-27 13:35:45.022 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20231204 2023-11-27 13:35:45.023 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:45.023 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:45.024 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20231204 values less than (1701734400000000)) 2023-11-27 13:35:45.261 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:45.262 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:45.263 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2023-11-27 13:35:45.269 INFO 10 --- [pool-heartbeat-metric-collect-10.120.0.1:8080-01,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from ASSIGNED to WORKING. 2023-11-27 13:35:45.269 INFO 10 --- [pool-heartbeat-metric-collect-10.120.0.1:8080-01,,] c.o.o.server.config.MetricConfiguration : Consumer had rebalanced. 2023-11-27 13:35:45.282 INFO 10 --- [pool-heartbeat-metric-collect-10.120.0.1:8080-01,,] c.o.o.c.s.registry.ConsumerRegistry : Slice modified, module=metric-collect, name=10.120.0.1:8080-0, consumerId=2, resources=[2000111, 2000113, 2000115, 2000117, 2000122, 2000124, 2000126, 2000128, 2000133, 2000135, 2000137, 2000142, 2000144, 2000149, 2000151, 2000153, 2000158, 2000160, 2000180, 2000182, 2000184, 2000186, 2000188, 2000190, 2000192, 2000194] 2023-11-27 13:35:45.313 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2023-11-27 13:35:45.318 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20231205 2023-11-27 13:35:45.319 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:45.319 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:45.320 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20231205 values less than (1701820800000000)) 2023-11-27 13:35:45.325 INFO 10 --- [pool-heartbeat-metric-collect-10.120.0.1:8080-11,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from ASSIGNED to WORKING. 2023-11-27 13:35:45.325 INFO 10 --- [pool-heartbeat-metric-collect-10.120.0.1:8080-11,,] c.o.o.server.config.MetricConfiguration : Consumer had rebalanced. 2023-11-27 13:35:45.368 INFO 10 --- [pool-heartbeat-metric-collect-10.120.0.1:8080-11,,] c.o.o.c.s.registry.ConsumerRegistry : Slice modified, module=metric-collect, name=10.120.0.1:8080-1, consumerId=1, resources=[2000110, 2000112, 2000114, 2000116, 2000118, 2000123, 2000125, 2000127, 2000129, 2000134, 2000136, 2000141, 2000143, 2000145, 2000150, 2000152, 2000157, 2000159, 2000161, 2000181, 2000183, 2000185, 2000187, 2000189, 2000191, 2000193] 2023-11-27 13:35:45.448 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:35:45.458 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:35:45.541 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:45.541 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:45.543 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2023-11-27 13:35:45.585 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2023-11-27 13:35:45.594 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20231116 2023-11-27 13:35:45.599 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:45.599 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:45.600 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` drop partition (P20231116) 2023-11-27 13:35:45.674 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:45.674 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:45.677 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2023-11-27 13:35:45.689 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20231117 2023-11-27 13:35:45.690 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:45.690 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:45.691 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` drop partition (P20231117) 2023-11-27 13:35:45.785 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:45.785 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:45.786 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2023-11-27 13:35:45.817 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20231118 2023-11-27 13:35:45.832 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:45.832 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:45.838 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` drop partition (P20231118) 2023-11-27 13:35:46.044 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:46.044 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:46.045 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2023-11-27 13:35:46.080 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:46.080 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:46.081 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:35:46.083 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:46.083 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:46.084 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2023-11-27 13:35:46.109 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20231203 2023-11-27 13:35:46.117 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:46.117 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:46.121 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20231203 values less than (1701648000000000)) 2023-11-27 13:35:46.312 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:46.312 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:46.314 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2023-11-27 13:35:46.363 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2023-11-27 13:35:46.371 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20231204 2023-11-27 13:35:46.377 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:46.377 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:46.382 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20231204 values less than (1701734400000000)) 2023-11-27 13:35:46.541 INFO 10 --- [main,,] c.o.o.c.s.c.LocalSecurityConfiguration : Local JDBC authentication enabled: whiteList [/api/status.*, /api/v2/time, /api/v2/mc/info, /api/v2/loginKey, /api/v2/obproxy/loadBalanceInfo.*, .*Action=(ObRootServiceInfo|ObRootServiceRegister|ObIDCRegionInfo|GetObProxyConfig|AllocateClusterIdByNameAndIdx|DeleteObRootServiceInfoByClusterName|GetObRootServiceInfoUrlTemplate).*, performance/ash/reports/.*/data], basicAuth true, CSRF true 2023-11-27 13:35:46.541 WARN 10 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/assets/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2023-11-27 13:35:46.542 INFO 10 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/assets/**'] 2023-11-27 13:35:46.543 WARN 10 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/index.html']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2023-11-27 13:35:46.543 INFO 10 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/index.html'] 2023-11-27 13:35:46.543 WARN 10 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.js']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2023-11-27 13:35:46.543 INFO 10 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.js'] 2023-11-27 13:35:46.543 WARN 10 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.css']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2023-11-27 13:35:46.543 INFO 10 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.css'] 2023-11-27 13:35:46.543 WARN 10 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/webjars/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2023-11-27 13:35:46.543 INFO 10 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/webjars/**'] 2023-11-27 13:35:46.543 WARN 10 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.svg']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2023-11-27 13:35:46.543 INFO 10 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.svg'] 2023-11-27 13:35:46.551 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:46.551 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:46.552 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2023-11-27 13:35:46.587 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2023-11-27 13:35:46.589 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20231205 2023-11-27 13:35:46.591 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:46.591 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:46.592 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20231205 values less than (1701820800000000)) 2023-11-27 13:35:46.750 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:46.750 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:46.751 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2023-11-27 13:35:46.787 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2023-11-27 13:35:46.791 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20231109 2023-11-27 13:35:46.799 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:46.799 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:46.801 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` drop partition (P20231109) 2023-11-27 13:35:46.868 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:46.868 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:46.869 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2023-11-27 13:35:46.880 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20231110 2023-11-27 13:35:46.883 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:46.883 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:46.884 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` drop partition (P20231110) 2023-11-27 13:35:46.952 INFO 10 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will secure com.oceanbase.ocp.core.security.conf.BasicAuthenticationSecurityConfiguration$$Lambda$2074/1946733125@7061d5c2 with [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@31cf6aa2, org.springframework.security.web.context.SecurityContextPersistenceFilter@58b1e9c7, org.springframework.security.web.header.HeaderWriterFilter@5b4ce459, org.springframework.security.web.authentication.logout.LogoutFilter@20c0e21, com.oceanbase.ocp.core.security.filter.PreAuthenticationFilter@2afb3317, org.springframework.security.web.authentication.www.BasicAuthenticationFilter@179adea2, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@5b7cdecd, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@5873ebc2, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@271cefbd, org.springframework.security.web.session.SessionManagementFilter@35f7ebc1, org.springframework.security.web.access.ExceptionTranslationFilter@6e8c7ec0, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@78bd6c04] 2023-11-27 13:35:46.959 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:46.959 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:46.961 INFO 10 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will secure com.oceanbase.ocp.core.security.conf.AccessKeySecurityConfiguration$$Lambda$2079/518091881@1244c8b0 with [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@31741039, org.springframework.security.web.context.SecurityContextPersistenceFilter@5db566f, org.springframework.security.web.header.HeaderWriterFilter@328ad0ea, org.springframework.security.web.authentication.logout.LogoutFilter@5c76c0c7, com.oceanbase.ocp.core.security.filter.PreAuthenticationFilter@2afb3317, com.oceanbase.ocp.core.security.accesskey.AccessKeyAuthenticationFilter@2c5adb25, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@7eee8280, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@1a870037, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@29bfe521, org.springframework.security.web.session.SessionManagementFilter@774dc243, org.springframework.security.web.access.ExceptionTranslationFilter@3c18eace, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@72befbd4] 2023-11-27 13:35:46.965 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2023-11-27 13:35:46.973 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20231111 2023-11-27 13:35:46.974 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:46.974 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:46.975 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` drop partition (P20231111) 2023-11-27 13:35:47.055 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:47.055 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:47.057 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2023-11-27 13:35:47.072 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:47.072 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:47.075 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:35:47.075 INFO 10 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will secure any request with [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@1c309f01, org.springframework.security.web.context.SecurityContextPersistenceFilter@5d95154e, org.springframework.security.web.header.HeaderWriterFilter@1e677f26, org.springframework.security.web.csrf.CsrfFilter@7eb56086, org.springframework.security.web.authentication.logout.LogoutFilter@418edaa5, com.oceanbase.ocp.core.security.filter.PreAuthenticationFilter@2afb3317, org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter@6954f824, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@46c25499, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@63799d28, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@52ebd17b, org.springframework.security.web.session.SessionManagementFilter@4b839d3c, org.springframework.security.web.access.ExceptionTranslationFilter@473b9ee3, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@46bd3b8e] 2023-11-27 13:35:47.078 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:47.078 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:47.078 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2023-11-27 13:35:47.101 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20231203 2023-11-27 13:35:47.110 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:47.110 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:47.111 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20231203 values less than (1701648000000000)) 2023-11-27 13:35:47.284 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:47.284 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:47.285 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2023-11-27 13:35:47.494 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2023-11-27 13:35:47.509 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20231204 2023-11-27 13:35:47.521 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:47.521 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:47.528 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20231204 values less than (1701734400000000)) 2023-11-27 13:35:47.689 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:47.689 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:47.690 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2023-11-27 13:35:47.709 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2023-11-27 13:35:47.711 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20231205 2023-11-27 13:35:47.712 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:47.713 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:47.713 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20231205 values less than (1701820800000000)) 2023-11-27 13:35:47.844 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:47.844 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:47.845 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2023-11-27 13:35:47.882 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2023-11-27 13:35:47.884 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20231025 2023-11-27 13:35:47.890 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:47.891 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:47.892 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` drop partition (P20231025) 2023-11-27 13:35:47.953 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:47.953 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:47.954 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2023-11-27 13:35:47.965 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20231026 2023-11-27 13:35:47.966 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:47.966 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:47.967 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` drop partition (P20231026) 2023-11-27 13:35:48.030 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:48.030 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:48.031 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2023-11-27 13:35:48.046 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20231027 2023-11-27 13:35:48.049 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:48.049 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:48.050 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` drop partition (P20231027) 2023-11-27 13:35:48.117 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:48.117 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:48.118 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2023-11-27 13:35:48.137 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:48.137 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:48.138 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:35:48.139 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:48.139 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:48.140 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2023-11-27 13:35:48.149 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20231203 2023-11-27 13:35:48.151 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:48.151 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:48.151 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20231203 values less than (1701648000000000)) 2023-11-27 13:35:48.271 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:48.271 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:48.272 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2023-11-27 13:35:48.301 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2023-11-27 13:35:48.303 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20231204 2023-11-27 13:35:48.383 INFO 10 --- [pool-heartbeat-metric-collect-10.120.0.1:8080-11,,] c.o.o.c.s.registry.ConsumerRegistry : Slice modified, module=metric-collect, name=10.120.0.1:8080-1, consumerId=1, resources=[2000111, 2000112, 2000113, 2000117, 2000118, 2000124, 2000125, 2000126, 2000137, 2000143, 2000144, 2000145, 2000149, 2000150, 2000151, 2000152, 2000157, 2000158, 2000182, 2000183, 2000184, 2000188, 2000189, 2000190] 2023-11-27 13:35:48.412 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:48.412 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:48.413 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20231204 values less than (1701734400000000)) 2023-11-27 13:35:48.430 INFO 10 --- [pool-heartbeat-metric-collect-10.120.0.1:8080-01,,] c.o.o.c.s.registry.ConsumerRegistry : Slice modified, module=metric-collect, name=10.120.0.1:8080-0, consumerId=2, resources=[2000110, 2000114, 2000115, 2000116, 2000122, 2000123, 2000127, 2000128, 2000129, 2000133, 2000134, 2000135, 2000136, 2000141, 2000142, 2000153, 2000159, 2000160, 2000161, 2000180, 2000181, 2000185, 2000186, 2000187, 2000191, 2000192, 2000193, 2000194] 2023-11-27 13:35:48.460 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:35:48.465 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:35:48.539 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:48.539 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:48.540 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2023-11-27 13:35:48.564 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2023-11-27 13:35:48.566 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20231205 2023-11-27 13:35:48.570 INFO 10 --- [main,,] o.s.cloud.commons.util.InetUtils : Cannot determine local hostname 2023-11-27 13:35:48.570 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:48.570 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:48.572 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20231205 values less than (1701820800000000)) 2023-11-27 13:35:48.770 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:48.770 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:48.771 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2023-11-27 13:35:48.822 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2023-11-27 13:35:48.825 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20231109 2023-11-27 13:35:48.826 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:48.826 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:48.827 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` drop partition (P20231109) 2023-11-27 13:35:48.882 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:48.882 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:48.883 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2023-11-27 13:35:48.892 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20231110 2023-11-27 13:35:48.893 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:48.893 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:48.894 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` drop partition (P20231110) 2023-11-27 13:35:48.953 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:48.953 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:48.954 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2023-11-27 13:35:48.962 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20231111 2023-11-27 13:35:48.964 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:48.964 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:48.965 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` drop partition (P20231111) 2023-11-27 13:35:49.031 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:49.031 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:49.033 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2023-11-27 13:35:49.041 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:49.041 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:49.042 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:35:49.043 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:49.043 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:49.043 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2023-11-27 13:35:49.054 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20231203 2023-11-27 13:35:49.055 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:49.055 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:49.056 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20231203 values less than (1701648000000000)) 2023-11-27 13:35:49.166 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:49.166 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:49.168 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2023-11-27 13:35:49.195 INFO 10 --- [main,,] o.s.b.a.e.web.EndpointLinksResolver : Exposing 3 endpoint(s) beneath base path '/actuator' 2023-11-27 13:35:49.251 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2023-11-27 13:35:49.253 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20231204 2023-11-27 13:35:49.255 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:49.255 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:49.255 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20231204 values less than (1701734400000000)) 2023-11-27 13:35:49.410 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:49.410 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:49.411 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2023-11-27 13:35:49.461 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2023-11-27 13:35:49.463 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20231205 2023-11-27 13:35:49.464 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:49.464 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:49.464 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20231205 values less than (1701820800000000)) 2023-11-27 13:35:49.668 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:49.668 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:49.691 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2023-11-27 13:35:49.792 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2023-11-27 13:35:49.806 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20231109 2023-11-27 13:35:49.827 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:49.827 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:49.837 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` drop partition (P20231109) 2023-11-27 13:35:49.931 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:49.931 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:49.934 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2023-11-27 13:35:49.954 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20231110 2023-11-27 13:35:49.956 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:49.956 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:49.957 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` drop partition (P20231110) 2023-11-27 13:35:50.010 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:50.010 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:50.011 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2023-11-27 13:35:50.012 INFO 10 --- [main,,] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path '' 2023-11-27 13:35:50.031 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20231111 2023-11-27 13:35:50.032 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:50.032 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:50.033 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` drop partition (P20231111) 2023-11-27 13:35:50.075 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:50.075 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:50.076 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2023-11-27 13:35:50.096 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:50.096 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:50.097 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:35:50.098 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:50.099 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:50.099 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2023-11-27 13:35:50.108 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20231203 2023-11-27 13:35:50.109 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:50.109 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:50.110 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20231203 values less than (1701648000000000)) 2023-11-27 13:35:50.228 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:50.228 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:50.229 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2023-11-27 13:35:50.257 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2023-11-27 13:35:50.259 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20231204 2023-11-27 13:35:50.260 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:50.260 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:50.261 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20231204 values less than (1701734400000000)) 2023-11-27 13:35:50.398 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:50.398 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:50.400 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2023-11-27 13:35:50.429 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2023-11-27 13:35:50.431 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20231205 2023-11-27 13:35:50.432 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:50.432 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:50.437 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20231205 values less than (1701820800000000)) 2023-11-27 13:35:50.566 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:50.567 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:50.567 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2023-11-27 13:35:50.609 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2023-11-27 13:35:50.620 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20231116 2023-11-27 13:35:50.621 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:50.621 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:50.623 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` drop partition (P20231116) 2023-11-27 13:35:50.681 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:50.681 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:50.682 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2023-11-27 13:35:50.689 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20231117 2023-11-27 13:35:50.690 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:50.690 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:50.691 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` drop partition (P20231117) 2023-11-27 13:35:50.745 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:50.745 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:50.746 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2023-11-27 13:35:50.753 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20231118 2023-11-27 13:35:50.754 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:50.754 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:50.755 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` drop partition (P20231118) 2023-11-27 13:35:50.867 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:50.867 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:50.868 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2023-11-27 13:35:50.882 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:50.882 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:50.884 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:35:50.895 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:50.895 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:50.905 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2023-11-27 13:35:50.914 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20231203 2023-11-27 13:35:50.916 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:50.916 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:50.917 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20231203 values less than (1701648000000000)) 2023-11-27 13:35:51.016 INFO 10 --- [main,,] o.s.cloud.commons.util.InetUtils : Cannot determine local hostname 2023-11-27 13:35:51.279 INFO 10 --- [http-nio-0.0.0.0-8080-exec-1,,] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring DispatcherServlet 'dispatcherServlet' 2023-11-27 13:35:51.279 INFO 10 --- [http-nio-0.0.0.0-8080-exec-1,,] o.s.web.servlet.DispatcherServlet : Initializing Servlet 'dispatcherServlet' 2023-11-27 13:35:51.285 INFO 10 --- [http-nio-0.0.0.0-8080-exec-1,,] o.s.web.servlet.DispatcherServlet : Completed initialization in 6 ms 2023-11-27 13:35:51.297 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:51.297 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:51.298 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2023-11-27 13:35:51.349 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2023-11-27 13:35:51.351 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20231204 2023-11-27 13:35:51.354 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:51.354 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:51.355 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20231204 values less than (1701734400000000)) 2023-11-27 13:35:51.466 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:35:51.473 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:35:51.519 INFO 10 --- [pool-ocp-schedules-1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2023-11-27 13:35:51.569 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:51.569 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:51.570 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2023-11-27 13:35:51.572 INFO 10 --- [pool-ocp-schedules-12,,7de27d0bf86e] .s.a.AnnotationAsyncExecutionInterceptor : More than one TaskExecutor bean found within the context, and none is named 'taskExecutor'. Mark one of them as primary or name it 'taskExecutor' (possibly as an alias) in order to use it for async processing: [alarmTaskExecutor, notificationSendExecutor, alarmDetectMetricExecutor, overviewContainerExecutor, overviewServiceExecutor] 2023-11-27 13:35:51.596 INFO 10 --- [pool-ocp-schedules-2,,d4ad29d85736] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-11-27 13:35:51.629 INFO 10 --- [pool-ocp-schedules-13,,949a3cc9ee6d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:35:51.634 INFO 10 --- [pool-ocp-schedules-11,,6d1f3280bb58] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=0 2023-11-27 13:35:51.634 INFO 10 --- [pool-ocp-schedules-1,,e70cd7c90cd4] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2023-11-27 13:35:51.684 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2023-11-27 13:35:51.695 INFO 10 --- [pool-ocp-schedules-15,,cc367e670d96] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-11-27 13:35:51.697 INFO 10 --- [pool-ocp-schedules-10,,a46f92987ca3] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":0,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":0,"secondWriteQueue":0,"minuteWriteCacheSize":0} 2023-11-27 13:35:51.697 INFO 10 --- [pool-ocp-schedules-10,,a46f92987ca3] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{} 2023-11-27 13:35:51.729 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20231205 2023-11-27 13:35:51.744 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:51.745 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:51.748 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20231205 values less than (1701820800000000)) 2023-11-27 13:35:51.754 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ************************* ACTIVE OCP PROPERTIES ****************************** 2023-11-27 13:35:51.754 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : Active profiles: [composite, default] 2023-11-27 13:35:51.759 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : apsara.audit.enabled=false 2023-11-27 13:35:51.759 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : backup.dbname.prefix=backup 2023-11-27 13:35:51.759 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : display-parse-failed-sql=true 2023-11-27 13:35:51.759 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : enable-sql.parse-service=true 2023-11-27 13:35:51.759 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : logging.file=/home/admin/ocp-server/log/ocp.log 2023-11-27 13:35:51.759 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : logging.file.clean-history-on-start=1 2023-11-27 13:35:51.759 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : logging.file.max-history=100 2023-11-27 13:35:51.760 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : logging.file.max-size=100MB 2023-11-27 13:35:51.760 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : logging.file.name=/home/admin/ocp-server/log/ocp.log 2023-11-27 13:35:51.760 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : logging.file.total-size-cap=10GB 2023-11-27 13:35:51.760 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : logging.level.com.oceanbase.ocp=INFO 2023-11-27 13:35:51.760 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : logging.level.org.hibernate.SQL=INFO 2023-11-27 13:35:51.760 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : logging.level.web=INFO 2023-11-27 13:35:51.760 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : logging.pattern.console=%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(%5p) %clr(10){magenta} %clr(---){faint} %clr([%t,%X{traceId},%X{spanId}]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m%n%wEx 2023-11-27 13:35:51.760 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : logging.pattern.file=%d{yyyy-MM-dd HH:mm:ss.SSS} %5p 10 --- [%t,%X{traceId},%X{spanId}] %-40.40logger{39} : %m%n%wEx 2023-11-27 13:35:51.760 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : mc.ocp.cluster.role.sync.check-timeout-seconds=120 2023-11-27 13:35:51.760 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : mgragent.log.level=info 2023-11-27 13:35:51.760 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : monagent.log.level=info 2023-11-27 13:35:51.760 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ob.logtailer.keyword.alarm.rules=[{"keyword":"partition table update task cost too much time to execute", "logLevel":"error", "svrType":"observer", "errorCode":4015, "alarmLevel":2}, {"keyword":"is_out_of_memstore_mem=true", "logLevel":"error", "svrType":"observer", "errorCode":100000, "alarmLevel":1}, {"keyword":"clog disk is almost full", "logLevel":"error", "svrType":"observer", "errorCode":100001, "alarmLevel":2}, {"keyword":"set_disk_error:attention!!!", "logLevel":"error", "svrType":"observer", "errorCode":100002, "alarmLevel":2}, {"keyword":"error occurs on data disk or slog disk", "logLevel":"warn", "svrType":"observer", "errorCode":100003, "alarmLevel":2}, {"keyword":"right_to_die_or_duty_to_live", "logLevel":"error", "svrType":"observer", "errorCode":100004, "alarmLevel":1}, {"keyword":"on_fatal_error", "logLevel":"error", "svrType":"observer", "errorCode":4070, "alarmLevel":1}] 2023-11-27 13:35:51.760 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : obsdk.connector.cleanup.period.seconds=300 2023-11-27 13:35:51.760 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : obsdk.connector.holder.capacity=100 2023-11-27 13:35:51.760 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : obsdk.connector.init.executor.thread-count=16 2023-11-27 13:35:51.760 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : obsdk.connector.init.timeout.millis=3000 2023-11-27 13:35:51.761 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : obsdk.connector.max-idle.seconds=3600 2023-11-27 13:35:51.761 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : obsdk.ob.connection.mode=proxy 2023-11-27 13:35:51.761 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : obsdk.operation.global.timeout.millis=300000 2023-11-27 13:35:51.761 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : obsdk.print.sql=true 2023-11-27 13:35:51.761 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : obsdk.slow.query.threshold.millis=1000 2023-11-27 13:35:51.761 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : obsdk.socket.connect.timeout.millis=2000 2023-11-27 13:35:51.761 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : obsdk.socket.read.timeout.millis=1800000 2023-11-27 13:35:51.761 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.agent.auth.metric-auth-enabled=true 2023-11-27 13:35:51.761 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.agent.auth.username=ocp_agent 2023-11-27 13:35:51.761 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.agent.home.path.prefix=/home/admin 2023-11-27 13:35:51.761 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.agent.manager.http.port=62888 2023-11-27 13:35:51.761 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.agent.monitor.http.port=62889 2023-11-27 13:35:51.761 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.agent.ob-cleaner.config={"ob_log":{"path":"{INSTALL_PATH}/log","disk_threshold":80,"rules":[{"file_reg":"\\\\([a-z]\\\\+\\\\.\\\\)\\\\?[a-z]\\\\+\\\\.log\\\\.[0-9]\\\\+","retention_days":8,"keep_percentage":0.6},{"file_reg":"\\\\([a-z]\\\\+\\\\.\\\\)\\\\?[a-z]\\\\+\\\\.log\\\\.wf\\\\.[0-9]\\\\+","retention_days":30,"keep_percentage":0.8}]},"core_log":{"path":"{INSTALL_PATH}","disk_threshold":80,"rules":[{"file_reg":"core\\\\.[0-9]\\\\+","retention_days":8,"keep_percentage":0.6}]}} 2023-11-27 13:35:51.761 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.channel.modify.enabled=true 2023-11-27 13:35:51.761 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.datasource.query-slow-sql-end-time-offset-seconds=130 2023-11-27 13:35:51.761 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.datasource.query-slow-sql-timeout-seconds=3 2023-11-27 13:35:51.763 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.datasource.slow-sql-circuit-breaker-seconds=300 2023-11-27 13:35:51.763 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.datasource.slow-sql-exclude-obclusters=sys#* 2023-11-27 13:35:51.763 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.datasource.slowsql-query-limit=2000 2023-11-27 13:35:51.763 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.datasource.trans-min-duration-seconds=60 2023-11-27 13:35:51.763 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.datasource.trans-min-log-size-mb=0.5 2023-11-27 13:35:51.763 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.datasource.trans-query-limit=2000 2023-11-27 13:35:51.763 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.datasource.trans-stat-circuit-breaker-seconds=300 2023-11-27 13:35:51.763 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.datasource.trans-stat-exclude-obclusters= 2023-11-27 13:35:51.763 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.datasource.trans.query-end-time-offset-seconds=15 2023-11-27 13:35:51.763 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.detect.too-many-targets-alarm-threshold=20 2023-11-27 13:35:51.764 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.enabled=true 2023-11-27 13:35:51.764 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.event.resend-interval-seconds=180 2023-11-27 13:35:51.764 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.host.property=svr_ip 2023-11-27 13:35:51.764 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.notification.archive-batch-count=3000 2023-11-27 13:35:51.764 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.notification.archive-once-times=100 2023-11-27 13:35:51.764 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.notification.expired-days=90 2023-11-27 13:35:51.764 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.query.metric-offset-seconds=10 2023-11-27 13:35:51.764 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.query.minute.metric-window-ratio=3 2023-11-27 13:35:51.764 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.query.second.metric-window-ratio=3 2023-11-27 13:35:51.764 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.query.sql.metric-limit=2000 2023-11-27 13:35:51.764 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.recover-message-template= 2023-11-27 13:35:51.764 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.resolve-timeout-minutes=7200 2023-11-27 13:35:51.764 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.send.batch-max=20 2023-11-27 13:35:51.764 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.send.enabled=true 2023-11-27 13:35:51.764 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.send.once-retry-times=1 2023-11-27 13:35:51.764 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.send.once-timeout-ms=10000 2023-11-27 13:35:51.764 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.send.period.ms=2000 2023-11-27 13:35:51.764 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.send.retry-timeout-minutes=60 2023-11-27 13:35:51.765 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.send.total-failed-retry-times=3 2023-11-27 13:35:51.765 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.send.total-timeout-seconds=60 2023-11-27 13:35:51.765 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.alarm.send.will-be-sent-in-period-seconds=1800 2023-11-27 13:35:51.765 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.analyze.enabled=false 2023-11-27 13:35:51.765 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.analyze.es.client.addresses=null 2023-11-27 13:35:51.765 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.analyze.es.client.password=null 2023-11-27 13:35:51.765 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.analyze.es.client.username=null 2023-11-27 13:35:51.765 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.analyze.es.index.keep.days=7 2023-11-27 13:35:51.765 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.analyze.es.index.replicas=1 2023-11-27 13:35:51.765 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.analyze.es.index.shards=12 2023-11-27 13:35:51.765 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.analyze.ob.log.enabled=false 2023-11-27 13:35:51.765 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.analyze.ob.trace.enabled=false 2023-11-27 13:35:51.766 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.audit.enabled=true 2023-11-27 13:35:51.766 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.backup.agent.relation.file.full-path.name=#{systemProperties['user.home'].concat('/ocp-server/etc/backup_agent_ob_relations_config.yaml')} 2023-11-27 13:35:51.766 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.backup.alarm.backup-data-retention-days=7 2023-11-27 13:35:51.766 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.backup.alarm.backup-liboblog-expire-days=7 2023-11-27 13:35:51.766 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.backup.backup-command.timeout=300 2023-11-27 13:35:51.766 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.backup.logical.log.meta.time.continuity.threshold.seconds=10 2023-11-27 13:35:51.766 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.backup.oss.support.cname=true 2023-11-27 13:35:51.766 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.backup.restore-time-pre-check.enabled=true 2023-11-27 13:35:51.766 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.backup.storage.collect.command.timeout=3600000 2023-11-27 13:35:51.766 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.backup.storage.collect.host.task.concurrency=8 2023-11-27 13:35:51.766 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.backup.storage.collect.max.retry.times=2 2023-11-27 13:35:51.766 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.backup.storage.collect.obtain-idle-host-interval-seconds=60 2023-11-27 13:35:51.766 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.backup.storage.collect.obtain-idle-host-times=60 2023-11-27 13:35:51.766 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.backup.storage.collect.period.minutes=180 2023-11-27 13:35:51.766 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.backup.storage.collect.schedule.enabled=true 2023-11-27 13:35:51.766 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.backup.storage.collect.task.timeout=10800000 2023-11-27 13:35:51.767 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.cloud.data.platform.site.url=null 2023-11-27 13:35:51.767 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.cloud.meta.platform.site.url=null 2023-11-27 13:35:51.767 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.cluster.id=1 2023-11-27 13:35:51.767 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.cluster.name=ocp-cluster 2023-11-27 13:35:51.767 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.cluster.obVersion.parameter.template.relation=[{"versionList":["2.2.77"],"parameterTemplateName":"2.2.77 Version Parameter Template"}] 2023-11-27 13:35:51.767 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.cluster.role=FOLLOWER 2023-11-27 13:35:51.768 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.config-url.site.url= 2023-11-27 13:35:51.768 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.config-url.use-request-address=true 2023-11-27 13:35:51.768 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.debug.web.cdn.enabled=false 2023-11-27 13:35:51.768 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.debug.web.cdn.location=https://gw-office.alipayobjects.com/a-sit/g/oceanbase/ocp-v2/2.5.0 2023-11-27 13:35:51.768 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.distributed.server.expire.seconds=30 2023-11-27 13:35:51.768 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.distributed.server.update.seconds=10 2023-11-27 13:35:51.768 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.distributed.service.expire.seconds={CMDB_SYNCER:30, TASK_MANAGER:30, RATE_LIMIT:30, PERF_SQL_ROLLUP:30, SQL_INDEX_DIAGNOSER:30, SQL_PERF_DIAGNOSER:30, PLAN_CHANGE_DETECTOR:30, MC_MANAGER:30} 2023-11-27 13:35:51.768 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.distributed.service.update.seconds={CMDB_SYNCER:10, TASK_MANAGER:10, RATE_LIMIT:10, PERF_SQL_ROLLUP:15, SQL_INDEX_DIAGNOSER:30, SQL_PERF_DIAGNOSER:30, PLAN_CHANGE_DETECTOR:30, MC_MANAGER:10} 2023-11-27 13:35:51.769 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.env=private 2023-11-27 13:35:51.769 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.event.audit.apsara.sls.access-key=****** 2023-11-27 13:35:51.769 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.event.audit.apsara.sls.access-secret=****** 2023-11-27 13:35:51.769 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.event.audit.apsara.sls.endpoint=TO_BE_REPLACED 2023-11-27 13:35:51.769 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.file.built-in.bucket=built-in 2023-11-27 13:35:51.769 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.file.default-block-split-size=1048576 2023-11-27 13:35:51.769 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.file.file-server.bucket.type=LOCAL 2023-11-27 13:35:51.769 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.file.local.built-in.dir=#{systemProperties['user.home'].concat('/ocp-server/lib')} 2023-11-27 13:35:51.769 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.file.local.dir=#{systemProperties['user.home'].concat('/data/files')} 2023-11-27 13:35:51.769 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.file.max-concurrent-count=16 2023-11-27 13:35:51.769 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.file.oss-block-split-size=1048576 2023-11-27 13:35:51.769 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.file.supported-architecture=x86_64,AArch64 2023-11-27 13:35:51.769 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.file.sys-package.bucket=sys-package 2023-11-27 13:35:51.769 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.file.try-lock-timeout-milliseconds=60000 2023-11-27 13:35:51.769 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.file.upload.sha1.whitelist=94c5350f98ba9dbb847f4943bb7bf33190d8003c,e9cadbc17aa496eab459bcbc7e4df213727dc766 2023-11-27 13:35:51.769 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.health-check.config.threshold={"-1":{"host_load":{"load1_per_cpu":2},"host_disk_status":{"host_disk_used_percent":85},"host_time":{"clock_diff":500,"ssh_interval":1000},"ob_server_disk_status":{"observer_disk_threshold_percentage":80},"ob_server_partition_count":{"ob_server_partition_count":100000},"obproxy_connection_count":{"obproxy_session_count_percentage":80}}} 2023-11-27 13:35:51.769 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.health-check.status.available-keyword=okay 2023-11-27 13:35:51.769 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.health-check.status.unavailable-keyword=bad 2023-11-27 13:35:51.769 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.host-manager.staragent.enable=false 2023-11-27 13:35:51.770 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.host.check.clock-diff.enable=false 2023-11-27 13:35:51.770 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.host.check.clock-diff.max-diff=50 2023-11-27 13:35:51.770 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.host.check.clock-diff.mode=0 2023-11-27 13:35:51.770 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.host.remote-command.default-timeout-millis=30000 2023-11-27 13:35:51.770 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.host.remote-command.min-timeout-millis=30000 2023-11-27 13:35:51.770 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.host.ssh-ping.max-delay-millis=3000 2023-11-27 13:35:51.770 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.account.system.active=false 2023-11-27 13:35:51.770 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.auth=local 2023-11-27 13:35:51.770 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.auth.access-key.enabled=true 2023-11-27 13:35:51.770 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.auth.apsara.aas.domain=TO_BE_REPLACED 2023-11-27 13:35:51.770 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.auth.apsara.aas.oauth-key=****** 2023-11-27 13:35:51.770 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.auth.apsara.aas.oauth-secret=****** 2023-11-27 13:35:51.770 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.auth.apsara.aas.oauth-url=TO_BE_REPLACED 2023-11-27 13:35:51.770 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.auth.apsara.aas.ops-site-domain=TO_BE_REPLACED 2023-11-27 13:35:51.770 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.auth.apsara.aas.ops-site-login-url=TO_BE_REPLACED 2023-11-27 13:35:51.770 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.auth.apsara.aas.ops-site-logout-url=TO_BE_REPLACED 2023-11-27 13:35:51.770 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.auth.apsara.aas.user-site-domain=TO_BE_REPLACED 2023-11-27 13:35:51.770 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.auth.apsara.aas.user-site-login-url=TO_BE_REPLACED 2023-11-27 13:35:51.771 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.auth.apsara.aas.user-site-logout-url=TO_BE_REPLACED 2023-11-27 13:35:51.771 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.auth.apsara.oam.access-key=****** 2023-11-27 13:35:51.771 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.auth.apsara.oam.access-secret=****** 2023-11-27 13:35:51.771 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.auth.apsara.oam.domain=TO_BE_REPLACED 2023-11-27 13:35:51.771 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.auth.apsara.oam.endpoint=TO_BE_REPLACED 2023-11-27 13:35:51.771 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.auth.apsara.oam.region-name=TO_BE_REPLACED 2023-11-27 13:35:51.771 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.auth.apsara.ram.endpoint=TO_BE_REPLACED 2023-11-27 13:35:51.771 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.auth.apsara.ram.region-name=TO_BE_REPLACED 2023-11-27 13:35:51.771 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.auth.basic.enabled=true 2023-11-27 13:35:51.771 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.auth.buc.admin-employee-ids=146380,073156,032575 2023-11-27 13:35:51.771 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.auth.buc.app-name=TO_BE_REPLACED 2023-11-27 13:35:51.771 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.auth.buc.login-env=daily 2023-11-27 13:35:51.771 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.auth.buc.login-url=TO_BE_REPLACED 2023-11-27 13:35:51.771 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.auth.buc.logout-url=TO_BE_REPLACED 2023-11-27 13:35:51.771 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.auth.whitelist=/api/v2/time,/api/v2/mc/info,/api/v2/loginKey,/api/v2/obproxy/loadBalanceInfo.*,.*Action=(ObRootServiceInfo|ObRootServiceRegister|ObIDCRegionInfo|GetObProxyConfig|AllocateClusterIdByNameAndIdx|DeleteObRootServiceInfoByClusterName|GetObRootServiceInfoUrlTemplate).*,performance/ash/reports/.*/data 2023-11-27 13:35:51.771 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.csrf.enabled=true 2023-11-27 13:35:51.771 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.csrf.url.excluded=/api/v2/iam/login.*,.*Action=(ObRootServiceInfo|ObRootServiceRegister|ObIDCRegionInfo|GetObProxyConfig|AllocateClusterIdByNameAndIdx|DeleteObRootServiceInfoByClusterName).*,/api/v2/monitor/metric/scan.* 2023-11-27 13:35:51.771 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.login.client.lockout-minutes=10 2023-11-27 13:35:51.771 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.login.client.max-attempts=10 2023-11-27 13:35:51.772 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.login.client.white-list=null 2023-11-27 13:35:51.772 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.login.lockout-minutes=30 2023-11-27 13:35:51.772 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.login.max-attempts=5 2023-11-27 13:35:51.772 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.preserved.username.list=admin,sys,system,sysdba,dba,oceanbase,obproxy,proxyro,ocp,ocp_monitor,ocp_agent,odc,oms,backup,drc,lbacsys,odp,temp 2023-11-27 13:35:51.772 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.rate-limit.enabled=true 2023-11-27 13:35:51.772 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.rate-limit.internal-api.connect.timeout=1000 2023-11-27 13:35:51.772 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.rate-limit.internal-api.read.timeout=1000 2023-11-27 13:35:51.772 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.rate-limit.on-ip.enabled=false 2023-11-27 13:35:51.772 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.rate-limit.on-user.enabled=true 2023-11-27 13:35:51.773 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.rate-limit.task.schedule.period=3 2023-11-27 13:35:51.773 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.role.forbidden.update.username.list=admin,system,obpass,obproxy,ocp_admin,monitor 2023-11-27 13:35:51.773 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.iam.user.access-key.max.count=5 2023-11-27 13:35:51.773 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.idempotent.client-token.expire.time=14d 2023-11-27 13:35:51.773 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.inspection.scrips.sequence-availability.threshold-percentage=0.8 2023-11-27 13:35:51.773 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.inspection.target-cluster-not-exists-name=__UNKNOWN__ 2023-11-27 13:35:51.773 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.inspection.target-database-not-exists-name=__UNKNOWN__ 2023-11-27 13:35:51.773 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.inspection.target-host-not-exists-ip=__UNKNOWN__ 2023-11-27 13:35:51.773 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.inspection.target-tenant-not-exists-name=__UNKNOWN__ 2023-11-27 13:35:51.773 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.inspection.task.core-pool-size=5 2023-11-27 13:35:51.773 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.inspection.task.max-pool-size=100 2023-11-27 13:35:51.773 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.inspection.task.queue-size=100 2023-11-27 13:35:51.773 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.inspection.task.thread-keep-alive-time=100 2023-11-27 13:35:51.773 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.internal.rpc.user.password=****** 2023-11-27 13:35:51.773 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.internal.rpc.user.username=system 2023-11-27 13:35:51.773 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.log.download.http.connect.timeout=180000 2023-11-27 13:35:51.773 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.log.download.http.read.timeout=180000 2023-11-27 13:35:51.773 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.log.download.tmp.dir=#{systemProperties['user.home'].concat('/logs/ocp')} 2023-11-27 13:35:51.773 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.log.max-download-speed=1024 2023-11-27 13:35:51.774 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.log.regex.host=/var/log/message* 2023-11-27 13:35:51.774 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.log.regex.obproxy=/home/admin/logs/obproxy/log/* 2023-11-27 13:35:51.774 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.log.regex.observer=%s/log/* 2023-11-27 13:35:51.774 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.log.regex.ocp_agent=/home/admin/ocp_agent/log/* 2023-11-27 13:35:51.774 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.login.encryption.enabled=true 2023-11-27 13:35:51.774 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.login.encryption.private-key=****** 2023-11-27 13:35:51.774 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.login.encryption.public-key=****** 2023-11-27 13:35:51.774 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.meta.cluster.name=obcluster 2023-11-27 13:35:51.774 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.metric.collect.interval.second=5 2023-11-27 13:35:51.774 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.migration.meta.baseline=4.0.3-20230301 2023-11-27 13:35:51.774 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitor.collect.host.enabled=true 2023-11-27 13:35:51.774 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitor.collect.max-pool-size=3000 2023-11-27 13:35:51.774 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitor.collect.obcluster.enabled=true 2023-11-27 13:35:51.774 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitor.collect.obproxy.enabled=true 2023-11-27 13:35:51.774 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitor.collect.obsdk.sql-query-row-limit=10000 2023-11-27 13:35:51.774 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitor.data.retention-days={"ob_cluster_system_event":31,"ocp_metric_data_1":8,"ocp_metric_data_60":31,"metric_hour_data":1098,"metric_daily_data":1098} 2023-11-27 13:35:51.774 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitor.data.table-switch-time=1672704726 2023-11-27 13:35:51.774 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitor.datasource.query.timeout.sec=120 2023-11-27 13:35:51.774 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitor.datasource.socket.timeout.ms=120000 2023-11-27 13:35:51.774 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitor.exporter.check-active-period-seconds=60 2023-11-27 13:35:51.775 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitor.exporter.check-inactive-period-seconds=50 2023-11-27 13:35:51.775 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitor.exporter.inactive.threshold.seconds=300 2023-11-27 13:35:51.775 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitor.host.exporters=/metrics/node/host 2023-11-27 13:35:51.775 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitor.host.metrics-list=load1,load5,load15,cpu_percent,io_read,io_write,io_read_time,io_write_time,io_read_byte,io_write_byte,net_recv,net_send,memory_used,memory_buffers,memory_free,host_disk_used,disk_percent,net_recv_percent,load1_per_cpu,memory_percent,net_send_percent,disk_readonly_flag,ssd_wear_indicator,tcp_retrans_percent,host_partition_usage_percent,memory_cached,ob_log_disk_usage_percent,ob_data_disk_usage_percent,ob_install_disk_usage_percent 2023-11-27 13:35:51.775 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitor.minute.exporters=:62888/metrics/stat,:62889/metrics/stat,/metrics/ob/extra,/metrics/obcloud/extra 2023-11-27 13:35:51.775 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitor.ob.exporters=/metrics/node/ob,/metrics/ob/basic,/metrics/ob/extra 2023-11-27 13:35:51.775 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitor.obproxy.exporters=/metrics/node/obproxy,/metrics/obproxy 2023-11-27 13:35:51.775 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitor.second.exporters=/metrics/node/host,/metrics/node/ob,/metrics/ob/basic,/metrics/node/obproxy,/metrics/obproxy 2023-11-27 13:35:51.775 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitordb.database=monitor_database 2023-11-27 13:35:51.775 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitordb.driverClassName=com.oceanbase.jdbc.Driver 2023-11-27 13:35:51.775 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitordb.druid.initialSize=10 2023-11-27 13:35:51.775 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitordb.druid.keepAlive=true 2023-11-27 13:35:51.775 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitordb.druid.maxActive=200 2023-11-27 13:35:51.775 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitordb.druid.maxWait=2000 2023-11-27 13:35:51.775 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitordb.druid.minIdle=2 2023-11-27 13:35:51.775 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitordb.druid.name=monitordb-connect-pool 2023-11-27 13:35:51.775 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitordb.druid.testWhileIdle=true 2023-11-27 13:35:51.775 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitordb.druid.validationQuery=SELECT 1 FROM DUAL 2023-11-27 13:35:51.775 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitordb.host=133.197.206.97 2023-11-27 13:35:51.775 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitordb.password=****** 2023-11-27 13:35:51.775 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitordb.port=2883 2023-11-27 13:35:51.775 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.monitordb.username=monitor_user@monitor_tenant#obcluster 2023-11-27 13:35:51.775 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.multi-az.enabled=false 2023-11-27 13:35:51.776 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.multi-cluster.enabled=false 2023-11-27 13:35:51.776 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.multi-cluster.heartbeat.expire.seconds=60 2023-11-27 13:35:51.776 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.multi-cluster.heartbeat.interval.seconds=10 2023-11-27 13:35:51.776 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.multi-cluster.replication.interval.seconds=30 2023-11-27 13:35:51.776 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.multi-cluster.replication.properties.metadb.global-tables=[{"tableName":"mc_ocp_cluster", "mode":"FULL", "querySql":"select `id`, `name`, `role`, `address`, `metadb_config`, `version`, `status`, `last_heartbeat_time`, `create_time`, `update_time` from mc_ocp_cluster","writeSql":"replace into mc_ocp_cluster (`id`, `name`, `role`, `address`, `metadb_config`, `version`, `status`, `last_heartbeat_time`, `create_time`, `update_time`) values (:id, :name, :role, :address, :metadb_config, :version, :status, :last_heartbeat_time, :create_time, :update_time)"},{"tableName":"mc_common_sequence", "mode":"FULL", "querySql":"select `id`, `name`, `next_val`, `create_time`, `update_time` from mc_common_sequence","writeSql":"replace into mc_common_sequence (`id`, `name`, `next_val`, `create_time`, `update_time`) values (:id, :name, :next_val, :create_time, :update_time)"},{"tableName":"mc_ob_cluster", "mode": "FULL", "querySql":"select `ocp_cluster_id`, `ocp_ob_cluster_id`, `ob_cluster_name`, `ob_cluster_id`, `type`, `architecture`, `rootserver_json`, `ob_version`, `locker_id`, `lock_count`, `create_time`, `update_time` from mc_ob_cluster", "writeSql":"replace into mc_ob_cluster (`ocp_cluster_id`, `ocp_ob_cluster_id`, `ob_cluster_name`, `ob_cluster_id`, `type`, `architecture`, `rootserver_json`, `ob_version`, `locker_id`, `lock_count`, `create_time`, `update_time`) values (:ocp_cluster_id, :ocp_ob_cluster_id, :ob_cluster_name, :ob_cluster_id, :type, :architecture, :rootserver_json, :ob_version, :locker_id, :lock_count, :create_time, :update_time)"}, {"tableName":"mc_ocp_cluster_remote_call", "mode":"INCREMENTAL", "querySql":"select `uuid`, `caller_id`, `executor_id`, `task_type`, `request_type`, `context`, `arguments_json`, `output_json`, `max_retry_times`, `retry_interval`, `current_retry_time`, `timeout`, `status`, `start_time`, `finish_time`, `create_time`, `update_time` from mc_ocp_cluster_remote_call where update_time >= :INCREMENTAL_COLUMN_VALUE", "writeSql":"replace into mc_ocp_cluster_remote_call (`uuid`, `caller_id`, `executor_id`, `task_type`, `request_type`, `context`, `arguments_json`, `output_json`, `max_retry_times`, `retry_interval`, `current_retry_time`, `timeout`, `status`, `start_time`, `finish_time`, `create_time`, `update_time`) values (:uuid, :caller_id, :executor_id, :task_type, :request_type, :context, :arguments_json, :output_json, :max_retry_times, :retry_interval, :current_retry_time, :timeout, :status, :start_time, :finish_time, :create_time, :update_time)", "queryMaxRecordSql":"select max(update_time) from mc_ocp_cluster_remote_call"}, {"tableName":"mc_ocp_cluster_remote_call_instance", "mode":"INCREMENTAL", "querySql":"select `uuid`, `remote_call_uuid`, `caller_id`, `executor_id`, `server_ip_port`, `task_type`, `request_type`, `context`, `arguments_json`, `output_json`, `timeout`, `status`, `start_time`, `finish_time`, `create_time`, `update_time` from mc_ocp_cluster_remote_call_instance where update_time >= :INCREMENTAL_COLUMN_VALUE", "writeSql":"replace into mc_ocp_cluster_remote_call_instance (`uuid`, `remote_call_uuid`, `caller_id`, `executor_id`, `server_ip_port`, `task_type`, `request_type`, `context`, `arguments_json`, `output_json`, `timeout`, `status`, `start_time`, `finish_time`, `create_time`, `update_time`) values (:uuid, :remote_call_uuid, :caller_id, :executor_id, :server_ip_port, :task_type, :request_type, :context, :arguments_json, :output_json, :timeout, :status, :start_time, :finish_time, :create_time, :update_time)","queryMaxRecordSql":"select max(update_time) from mc_ocp_cluster_remote_call_instance"}] 2023-11-27 13:35:51.776 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.network.inter-connector.agent.socks-port=1080 2023-11-27 13:35:51.776 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.network.inter-connector.host-address=localhost 2023-11-27 13:35:51.776 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.network.inter-connector.socks-port=1080 2023-11-27 13:35:51.776 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.network.inter-connector.web-port=8088 2023-11-27 13:35:51.776 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.network.vpc.mode.enabled=false 2023-11-27 13:35:51.776 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.ob.cluster.ops.blacklist=obcluster 2023-11-27 13:35:51.777 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.ob.resource.cache.enable=0 2023-11-27 13:35:51.777 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.ob.resource.cache.refresh.thread-core-size=20 2023-11-27 13:35:51.777 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.ob.resource.cache.refresh.thread-max-size=200 2023-11-27 13:35:51.777 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.ob.slowsql.threshold=100000 2023-11-27 13:35:51.777 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.ob.upgrade.allow-downgrade=false 2023-11-27 13:35:51.777 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.ob.upgrade.dependencies.full-path.name=#{systemProperties['user.home'].concat('/ocp-server/etc/oceanbase_upgrade_dep.yml')} 2023-11-27 13:35:51.777 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.ob.upgrade.script.cmd.timeout=0 2023-11-27 13:35:51.777 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.obcloud.api.address= 2023-11-27 13:35:51.777 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.operation.executor.connector.cache.max-idle-seconds=3600 2023-11-27 13:35:51.777 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.operation.executor.http.connect-timeout=5000 2023-11-27 13:35:51.777 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.operation.executor.http.connection.max-per-route=100 2023-11-27 13:35:51.777 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.operation.executor.http.connection.max-pool-size=100 2023-11-27 13:35:51.777 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.operation.executor.http.read-timeout=30000 2023-11-27 13:35:51.777 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.operation.executor.http.socket-timeout=3000 2023-11-27 13:35:51.777 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.operation.executor.http.thread.core-pool-size=500 2023-11-27 13:35:51.777 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.operation.executor.http.thread.keep-alive-time=120000 2023-11-27 13:35:51.777 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.operation.executor.http.thread.max-pool-size=1500 2023-11-27 13:35:51.777 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.operation.executor.ssh.connect-timeout=10000 2023-11-27 13:35:51.778 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.operation.executor.ssh.default-command-timeout=10000 2023-11-27 13:35:51.778 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.operation.ob.delete-server.reserve-data-enabled=false 2023-11-27 13:35:51.778 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.operation.ob.resource.trend.max-point=1440 2023-11-27 13:35:51.778 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.operation.ob.server.skip-io-bench=false 2023-11-27 13:35:51.778 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.operation.ob.single-idc-for-zone-enabled=true 2023-11-27 13:35:51.778 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.operation.ob.sql-execute.enabled=false 2023-11-27 13:35:51.778 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.operation.ob.sql-execute.sql-type.whitelist=SELECT,SHOW 2023-11-27 13:35:51.778 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.operation.ob.tenant.allow-small-unit=true 2023-11-27 13:35:51.778 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.operation.ob.tenant.unit-spec-limit-rules=[{"minObVersion":"2.0.0","maxObVersion":"4.0.0.0","memoryLowerLimit":5},{"minObVersion":"4.0.0.0","memoryLowerLimit":5}] 2023-11-27 13:35:51.778 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.operation.ob.tenant.whitelist.attach-default-whitelist=true 2023-11-27 13:35:51.778 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.operation.ob.unit.unused.max-reserve-hour=1 2023-11-27 13:35:51.778 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.operation.obproxy.readonly.default.password=****** 2023-11-27 13:35:51.778 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.operation.obproxy.readonly.default.username=proxyro 2023-11-27 13:35:51.778 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.operation.obproxy.start.command.template=./bin/obproxy -p 2883 -n %s -o obproxy_config_server_url='%s/services?User_ID=alibaba&UID=test&Action=GetObProxyConfig',enable_metadb_used=false,enable_strict_kernel_release=false,enable_proxy_scramble=true,proxy_mem_limited=2G,log_dir_size_threshold=10G,prometheus_sync_interval=1s 2023-11-27 13:35:51.778 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.operation.odp.root.default.password=****** 2023-11-27 13:35:51.778 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.overview.alarm.status.refresh.interval=10000 2023-11-27 13:35:51.778 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.overview.cluster.status.refresh.interval=10000 2023-11-27 13:35:51.778 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.overview.host.status.refresh.interval=10000 2023-11-27 13:35:51.780 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.overview.obproxy.status.refresh.interval=10000 2023-11-27 13:35:51.780 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.overview.tasks.status.refresh.interval=10000 2023-11-27 13:35:51.780 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.overview.tenant.status.refresh.interval=10000 2023-11-27 13:35:51.780 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.ash.earliest-analyse-time=8 2023-11-27 13:35:51.780 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.ash.max-analyse-range=60m 2023-11-27 13:35:51.780 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.ash.report-retention-days=90 2023-11-27 13:35:51.780 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.big.transaction.log.threshold.byte=524288 2023-11-27 13:35:51.780 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.big.transaction.time.threshold.ms=500 2023-11-27 13:35:51.780 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.collect.metadb.enabled=false 2023-11-27 13:35:51.780 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.parallel.sql.limit=2000 2023-11-27 13:35:51.780 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.parallel.sql.partition.retention.days=15 2023-11-27 13:35:51.780 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.snapshot.generation.enable=1 2023-11-27 13:35:51.781 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.snapshot.generation.white-list=null 2023-11-27 13:35:51.781 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql-diag.awful-performance-index-used-config={"enabled":true, "cpuTimeLimitUs":100000, "execPsLimit":5.0, "fullLogicalReadsLimit":10000} 2023-11-27 13:35:51.781 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql-diag.cpu-time-proportion-high-config={"enabled":true, "cpuTimeLimitUs":10000, "execPsLimit":10.0, "maxCpuTimeLimitUs":30000, "affectedRowsLimit":100, "execLimit":30, "sqlCountLimit":20, "cpuTimePercentLimit":20} 2023-11-27 13:35:51.781 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql-diag.database-white-list=oceanbase,information_schema,mysql,__recyclebin,sys,__public 2023-11-27 13:35:51.781 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql-diag.diagnose.standby-cluster.enabled=0 2023-11-27 13:35:51.781 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql-diag.execution-spike-config={"enabled":true, "cpuTimeLimitUs":3000, "execPsLimit":5.0, "pointsLimit":5, "compareMin":20, "avgExecutionMultiply":{"0":6, "1":5, "5":4, "10":3, "50": 2.5, "100": 2, "500": 1.6}, "stdExecutionMultiply":{"0":80, "1":60 , "3": 30,"5":20, "10":15, "20":8, "30":6, "50":5, "80":4}} 2023-11-27 13:35:51.781 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql-diag.index-diagnoser-config={"enabled":true,"schedulePeriodMin":5, "coreThreadSize":8, "maxThreadSize":16,"maxQueueSize":10000,"diagPeriodSec":300,"diagOffsetSec":60,"maxDiagPeriodSec":1800} 2023-11-27 13:35:51.781 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql-diag.ineffective-hint-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":20000} 2023-11-27 13:35:51.781 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql-diag.outline-history-query-days=30 2023-11-27 13:35:51.781 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql-diag.perf-diagnoser-config={"enabled":true,"schedulePeriodMin":5, "coreThreadSize":8, "maxThreadSize":16,"maxQueueSize":10000,"diagPeriodSec":300,"diagOffsetSec":60,"maxDiagPeriodSec":1800} 2023-11-27 13:35:51.781 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql-diag.performance-degradation-after-plan-changed-config={"enabled":true, "compareMin":5, "cpuTimeMultiply":{"0":50, "1":30, "10":10, "100":8, "1000":6}} 2023-11-27 13:35:51.781 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql-diag.performance-degradation-config={"enabled":true, "cpuTimeLimitUs":10000, "execPsLimit":5.0, "maxCpuTimeLimitUs":30000, "affectedRowsLimit":100, "execLimit":5, "compareMin":20, "cpuTimeMultiply":{"0":50, "1":30, "10":10, "100":8, "1000":6}} 2023-11-27 13:35:51.781 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql-diag.query-timeout=30000000 2023-11-27 13:35:51.782 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql-diag.row-lock-contention-high-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":100000, "elapsedTimeLimitUs":0} 2023-11-27 13:35:51.782 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql-diag.sql-diagnosis-retention-days=30 2023-11-27 13:35:51.782 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql-diag.table-scan-index-not-exists-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":20000} 2023-11-27 13:35:51.782 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql-diag.table-scan-index-not-used-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":20000} 2023-11-27 13:35:51.782 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql-diag.tenant-white-list=null 2023-11-27 13:35:51.782 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.advisor-plan-max-query-range=7d 2023-11-27 13:35:51.782 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.diag-plan-change-config={"enabled":true,"schedulePeriodMin":2, "coreThreadSize":8, "maxThreadSize":16,"maxQueueSize":500,"diagPeriodSec":120,"diagOffsetSec":120,"maxDiagPeriodSec":600} 2023-11-27 13:35:51.782 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.affected-rows-threshold=50000 2023-11-27 13:35:51.782 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.core-pool-size=30 2023-11-27 13:35:51.782 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-period-timeUs=1800000000 2023-11-27 13:35:51.782 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-pool-size=300 2023-11-27 13:35:51.782 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-queue-size=1000000 2023-11-27 13:35:51.782 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-thread-pool-task-limit=50000 2023-11-27 13:35:51.782 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.partition-count-threshold=2000 2023-11-27 13:35:51.782 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.return-rows-threshold=50000 2023-11-27 13:35:51.783 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.diag.sql-audit-stat-queue-max-size=1000000 2023-11-27 13:35:51.783 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.diag.sql-audit-stat-worker-core-size=30 2023-11-27 13:35:51.783 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.diag.sql-audit-stat-worker-max-size=300 2023-11-27 13:35:51.783 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.earliest-query-time=8d 2023-11-27 13:35:51.783 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.high-risk-inspect-service.enable=1 2023-11-27 13:35:51.783 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.high-risk-max-query-range=7d 2023-11-27 13:35:51.783 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.index-advisor-cpu-time-threshold=10 2023-11-27 13:35:51.783 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.index-advisor-elapsed-time-threshold=20 2023-11-27 13:35:51.783 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.info.parameterize.sql=false 2023-11-27 13:35:51.783 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.info.parse.sql.type=true 2023-11-27 13:35:51.783 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.max-query-range=24h 2023-11-27 13:35:51.783 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.new-sql.limit=2000 2023-11-27 13:35:51.783 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.new-sql.query.compare.interval.days=7 2023-11-27 13:35:51.783 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.new-sql.query.interval.days=7d 2023-11-27 13:35:51.783 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.parse.batch.insert.size=100 2023-11-27 13:35:51.783 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.parse.core.thread=8 2023-11-27 13:35:51.783 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.parse.insert.query.rate.limit=10 2023-11-27 13:35:51.783 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.parse.max.thread=16 2023-11-27 13:35:51.783 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.parse.query.limit=2000 2023-11-27 13:35:51.784 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.parse.queue.size=20000 2023-11-27 13:35:51.784 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.plan-hist-level0-granularity=30s 2023-11-27 13:35:51.784 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.plan-hist-level0-query-interval=2h 2023-11-27 13:35:51.784 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.plan-hist-level0-retention=8d 2023-11-27 13:35:51.784 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.prepare-partition-ahead=8 2023-11-27 13:35:51.784 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.query-digest-limit=0 2023-11-27 13:35:51.784 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.query-digest-trend.points=15 2023-11-27 13:35:51.784 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.query-timeout=30000000 2023-11-27 13:35:51.784 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.query.high-risk-sql.sql-text-short-length=50 2023-11-27 13:35:51.784 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.slow-sql-limit=2000 2023-11-27 13:35:51.784 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.sql-audit-sample-retention=8d 2023-11-27 13:35:51.784 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.sql-hist-level0-granularity=30s 2023-11-27 13:35:51.784 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.sql-hist-level0-query-interval=2h 2023-11-27 13:35:51.784 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.sql-hist-level0-retention=2d 2023-11-27 13:35:51.784 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.sql-hist-level1-granularity=2m 2023-11-27 13:35:51.784 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.sql-hist-level1-query-interval=12h 2023-11-27 13:35:51.784 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.sql-hist-level1-retention=8d 2023-11-27 13:35:51.784 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.sql-hist-level2-granularity=10m 2023-11-27 13:35:51.785 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.sql-hist-level2-query-interval=48h 2023-11-27 13:35:51.785 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.sql-hist-level2-retention=15d 2023-11-27 13:35:51.785 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.sql-info-secondary-cache-expiration=30m 2023-11-27 13:35:51.785 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.sql-query-digest-level0-query-interval=30m 2023-11-27 13:35:51.785 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.sql-query-digest-level1-query-interval=2h 2023-11-27 13:35:51.785 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.top-sql-limit=2000 2023-11-27 13:35:51.785 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.sql.tuning-advisor-cpu-time-ratio-of-plan-comparison=0.8 2023-11-27 13:35:51.785 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.transaction.detail.query.range.minute=30 2023-11-27 13:35:51.785 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.transaction.interval.second=5 2023-11-27 13:35:51.785 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.transaction.long.transaction.threshold.second=60 2023-11-27 13:35:51.785 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.transaction.partition.retention.days=7 2023-11-27 13:35:51.785 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.transaction.pending.transaction.threshold.second=60 2023-11-27 13:35:51.785 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.transaction.query.sql.sample.timeout.second=30 2023-11-27 13:35:51.785 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.transaction.sql.count.limit=10000 2023-11-27 13:35:51.785 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.workload.generate-report-thread-count=5 2023-11-27 13:35:51.785 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.perf.workload.report.tenant-top-sql-limit=10 2023-11-27 13:35:51.785 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.self.metrics.collect.enable=true 2023-11-27 13:35:51.785 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.site.url=http://133.197.206.97:8080 2023-11-27 13:35:51.786 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.startup-check.metadb.variable=[{"variableName": "max_allowed_packet", "operator": ">=", "targetValue": 4194304}, {"variableName": "ob_read_consistency", "operator": "=", "targetValue": "STRONG"}] 2023-11-27 13:35:51.786 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.startup-check.monitordb.variable=[{"variableName": "max_allowed_packet", "operator": ">=", "targetValue": 4194304}, {"variableName": "ob_read_consistency", "operator": "=", "targetValue": "STRONG"}] 2023-11-27 13:35:51.786 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.system.default.language=zh-CN 2023-11-27 13:35:51.786 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.system.default.timezone= 2023-11-27 13:35:51.786 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.system.event.excluded={balancer: {'start_sql_bkgd_task','finish_sql_bkgd_task'}} 2023-11-27 13:35:51.786 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.system.obproxy.metadb.cluster-name=obdv1 2023-11-27 13:35:51.786 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.system.obproxy.metadb.database=obproxy 2023-11-27 13:35:51.786 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.system.obproxy.metadb.enabled=false 2023-11-27 13:35:51.786 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.system.obproxy.metadb.sensitive-info={"configuredInfo":{"antmeta":{"root@sys":"9f6e186723de2f4873cad637e0e8b6e9"},"obdv1":{"root@obproxy":"8f3bab5c8face32f3a7da34de0773c9b"}},"defaultInfo":{"default":"899f7ad059fc3a3c19dee85886a8f45a7e67dd4896dbeb93"}} 2023-11-27 13:35:51.786 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.system.obproxy.metadb.user=root@obproxy 2023-11-27 13:35:51.786 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.table-cleaning.batch-interval-millis=10 2023-11-27 13:35:51.786 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.table-cleaning.batch-size=1000 2023-11-27 13:35:51.786 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.table-cleaning.preserve-days=30 2023-11-27 13:35:51.786 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.table-cleaning.thread.core-pool-size=4 2023-11-27 13:35:51.786 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.table-cleaning.thread.maximum-pool-size=16 2023-11-27 13:35:51.786 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.version=4.0.3 2023-11-27 13:35:51.786 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.version.full=4.0.3-20230301 2023-11-27 13:35:51.786 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ocp.web.tracking.enabled=True 2023-11-27 13:35:51.786 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : odp.client.automatic-match-work-thread=false 2023-11-27 13:35:51.786 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : odp.client.enable-strict-kernel-release=false 2023-11-27 13:35:51.787 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : odp.client.log-dir-size-threshold=1G 2023-11-27 13:35:51.787 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : odp.client.proxy-mem-limited=1G 2023-11-27 13:35:51.787 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : odp.client.syslog-level=INFO 2023-11-27 13:35:51.787 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : odp.client.work-thread-num=8 2023-11-27 13:35:51.789 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : okta.oauth2.client-id=TO_BE_REPLACED 2023-11-27 13:35:51.789 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : okta.oauth2.client-secret=****** 2023-11-27 13:35:51.789 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : okta.oauth2.issuer=https://example.com/oauth2/default 2023-11-27 13:35:51.789 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : okta.oauth2.redirect-uri=TO_BE_REPLACED 2023-11-27 13:35:51.789 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : perf.sql.top-sql-comparison.elapsed-time-increase-limit=0.2 2023-11-27 13:35:51.789 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : perf.sql.top-sql-comparison.error-percentage-increase-limit=0.1 2023-11-27 13:35:51.789 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : perf.sql.top-sql-comparison.executions-increase-limit=0.2 2023-11-27 13:35:51.789 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : perf.sql.top-sql-comparison.new-in-top-n-limit=100 2023-11-27 13:35:51.789 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : perf.sql.top-sql-comparison.remote-plan-percentage-increase-limit=0.1 2023-11-27 13:35:51.789 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : perf.sql.top-sql-comparison.return-rows-increase-limit=0.2 2023-11-27 13:35:51.790 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : server.address=0.0.0.0 2023-11-27 13:35:51.790 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : server.port=8080 2023-11-27 13:35:51.790 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : server.servlet.session.timeout=30m 2023-11-27 13:35:51.790 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : server.ssl.enabled=false 2023-11-27 13:35:51.790 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : server.ssl.key-alias=null 2023-11-27 13:35:51.790 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : server.ssl.key-password=null 2023-11-27 13:35:51.790 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : server.ssl.key-store= 2023-11-27 13:35:51.790 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : server.ssl.key-store-password=****** 2023-11-27 13:35:51.790 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : server.ssl.key-store-type=PKCS12 2023-11-27 13:35:51.790 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : slow-sql.query-sql-info-from.cache.directly=false 2023-11-27 13:35:51.790 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : spring.data.web.pageable.one-indexed-parameters=true 2023-11-27 13:35:51.790 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : spring.datasource.druid.initialSize=10 2023-11-27 13:35:51.791 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : spring.datasource.druid.keepAlive=true 2023-11-27 13:35:51.791 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : spring.datasource.druid.maxActive=100 2023-11-27 13:35:51.791 INFO 10 --- [pool-ocp-schedules-3,,7f560f5a418f] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-11-27 13:35:51.791 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : spring.datasource.druid.maxWait=2000 2023-11-27 13:35:51.791 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : spring.datasource.druid.minIdle=2 2023-11-27 13:35:51.791 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : spring.datasource.druid.name=metadb-connect-pool 2023-11-27 13:35:51.791 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : spring.datasource.druid.testWhileIdle=true 2023-11-27 13:35:51.791 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : spring.datasource.druid.validationQuery=SELECT 1 FROM DUAL 2023-11-27 13:35:51.791 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : spring.jpa.hibernate.naming-strategy=org.hibernate.cfg.ImprovedNamingStrategy 2023-11-27 13:35:51.791 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.MySQL5Dialect 2023-11-27 13:35:51.791 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : spring.jpa.properties.hibernate.globally_quoted_identifiers=true 2023-11-27 13:35:51.791 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : spring.servlet.multipart.enabled=true 2023-11-27 13:35:51.791 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : spring.servlet.multipart.max-file-size=2048MB 2023-11-27 13:35:51.791 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : spring.servlet.multipart.max-request-size=2048MB 2023-11-27 13:35:51.791 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : springdoc.api-docs.enabled=false 2023-11-27 13:35:51.791 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : springdoc.api-docs.path=/doc/api-docs 2023-11-27 13:35:51.791 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : springdoc.paths-to-exclude=null 2023-11-27 13:35:51.883 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : springdoc.paths-to-match=/api/v2/** 2023-11-27 13:35:51.883 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : springdoc.swagger-ui.path=/doc/swagger-ui.html 2023-11-27 13:35:51.883 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : sql-stat.flush-sql-info-to-cache=true 2023-11-27 13:35:51.883 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : storage.capacity.compute.executor.core-pool-size=8 2023-11-27 13:35:51.883 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : storage.capacity.compute.executor.max-pool-size=64 2023-11-27 13:35:51.905 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : storage.capacity.compute.executor.task-queue-size=1800 2023-11-27 13:35:51.905 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : top-sql.query-sql-info-from.cache.directly=false 2023-11-27 13:35:51.905 INFO 10 --- [main,,] c.o.ocp.server.trace.PropertyLogger : ****************************************************************************** 2023-11-27 13:35:51.837 INFO 10 --- [pool-ocp-schedules-2,,d4ad29d85736] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-11-27 13:35:51.914 INFO 10 --- [pool-ocp-schedules-14,,15f95356e9f3] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-11-27 13:35:52.002 INFO 10 --- [main,,] com.oceanbase.ocp.OcpServerApplication : Started OcpServerApplication in 166.548 seconds (JVM running for 168.141) 2023-11-27 13:35:52.013 INFO 10 --- [pool-ocp-schedules-15,,cc367e670d96] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-11-27 13:35:52.044 INFO 10 --- [pool-ocp-schedules-17,,9be614453cef] c.o.o.s.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=2 2023-11-27 13:35:52.045 INFO 10 --- [pool-ocp-schedules-3,,7f560f5a418f] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-11-27 13:35:52.052 INFO 10 --- [pool-ocp-schedules-14,,15f95356e9f3] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-11-27 13:35:52.177 INFO 10 --- [main,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT2M46.805879252S 2023-11-27 13:35:52.177 INFO 10 --- [main,,] c.o.o.b.spring.BootstrapRunListener : ocp is ready! 2023-11-27 13:35:52.189 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:52.189 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:52.191 INFO 10 --- [main,,] c.o.o.b.ext.ReinstallAgentAfterUpgrade : ReinstallAgentAfterUpgrade created 2023-11-27 13:35:52.192 INFO 10 --- [main,,] c.o.o.b.ext.ReinstallAgentAfterUpgrade : not UPGRADE, skip reinstall agent 2023-11-27 13:35:52.209 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2023-11-27 13:35:52.211 INFO 10 --- [pool-ocp-schedules-17,,9be614453cef] c.o.o.s.a.service.TaskRepositoryService : fetchNeedExecute, tryAcquireTasksCount=2, fetchedCount=2 2023-11-27 13:35:52.473 INFO 10 --- [pool-ocp-schedules-19,,eed126033239] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-11-27 13:35:52.475 INFO 10 --- [SimpleAsyncTaskExecutor-11,,] c.o.o.p.a.i.SessionEventSyncService : start to sync session wait event info, cluster ids is [2, 1] 2023-11-27 13:35:52.504 INFO 10 --- [pool-ocp-schedules-17,,9be614453cef] c.o.o.s.alarm.core.task.AlarmDetectTask : AlarmDetectTask created 2023-11-27 13:35:52.533 INFO 10 --- [pool-Check host status1,,16cff7425ff1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=133.197.206.97, originalStatus=ONLINE 2023-11-27 13:35:52.575 INFO 10 --- [pool-Check host status2,,daf979afdf7e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=133.197.204.3, originalStatus=OFFLINE 2023-11-27 13:35:52.586 INFO 10 --- [pool-Check host status3,,24738553ce85] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=133.197.206.92, originalStatus=AVAILABLE 2023-11-27 13:35:52.605 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2023-11-27 13:35:52.607 INFO 10 --- [pool-Check host status4,,075a09f98e62] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=133.197.204.10, originalStatus=OFFLINE 2023-11-27 13:35:52.611 INFO 10 --- [pool-Check host status5,,ee500bda988b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=133.197.204.11, originalStatus=OFFLINE 2023-11-27 13:35:52.615 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20231116 2023-11-27 13:35:52.616 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:52.616 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:52.621 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` drop partition (P20231116) 2023-11-27 13:35:52.624 INFO 10 --- [pool-Check host status6,,144f4493b486] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=6, ip=133.197.204.12, originalStatus=OFFLINE 2023-11-27 13:35:52.638 INFO 10 --- [pool-Check host status7,,77c9c57b6af2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=133.197.204.13, originalStatus=OFFLINE 2023-11-27 13:35:52.649 INFO 10 --- [pool-Check host status8,,904533283ee8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=9, ip=133.197.206.88, originalStatus=AVAILABLE 2023-11-27 13:35:52.663 INFO 10 --- [pool-ocp-schedules-17,,9be614453cef] c.o.o.s.a.utils.task.AlarmTaskManager : start task, taskInfo=TaskInfo(id=1, uuid=ab790a90-287a-4cb5-88b1-6f61d37cfe86, taskType=AlarmDetect, target=null, parameters=null, intervalSeconds=10, status=EXECUTING, executorName=10.120.0.1:10, createAt=2022-11-29T06:49:27.121153Z, startAt=2023-11-27T05:35:52.113Z, endAt=null, heartbeatAt=2023-11-27T05:35:52.113Z, result=executing) 2023-11-27 13:35:52.665 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : refresh host alarm wrapper 2023-11-27 13:35:52.747 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : refresh alarm target 2023-11-27 13:35:52.749 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdCleanService : HostIpSize=8, clusterSize=2, obproxyClusterSize=1 2023-11-27 13:35:52.790 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:35:52.843 INFO 10 --- [pool-ocp-schedules-9,,8ca48a74dead] c.o.o.s.a.c.aggregate.AggregatorFactory : createAggregator, channelId=100000, channelName=钉钉群消息样例 2023-11-27 13:35:52.843 INFO 10 --- [pool-ocp-schedules-9,,8ca48a74dead] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=100000 2023-11-27 13:35:52.906 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:52.906 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:52.908 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2023-11-27 13:35:52.974 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20231117 2023-11-27 13:35:53.006 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:53.006 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:53.019 INFO 10 --- [pool-ocp-schedules-17,,9be614453cef] c.o.o.s.a.c.aggregate.AggregatorFactory : createAggregator, channelId=100000, channelName=钉钉群消息样例 2023-11-27 13:35:53.019 INFO 10 --- [pool-ocp-schedules-17,,9be614453cef] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=100000 2023-11-27 13:35:53.043 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` drop partition (P20231117) 2023-11-27 13:35:53.051 INFO 10 --- [http-nio-0.0.0.0-8080-exec-1,791a900eff1e4b4d,b9cf6ce9844d] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.12, traceId=791a900eff1e4b4d, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:35:53.073 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.120.0.1:8080 2023-11-27 13:35:53.077 INFO 10 --- [http-nio-0.0.0.0-8080-exec-2,c299ce270b0e447b,acf8ead470b6] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.11, traceId=c299ce270b0e447b, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:35:53.122 INFO 10 --- [pool-Check host status1,,16cff7425ff1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:35:53.123 INFO 10 --- [pool-Check host status5,,ee500bda988b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-11-27 13:35:53.141 INFO 10 --- [pool-Check host status8,,904533283ee8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:35:53.149 INFO 10 --- [pool-Check host status6,,144f4493b486] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2023-11-27 13:35:53.177 INFO 10 --- [pool-Check host status2,,daf979afdf7e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:35:53.230 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:35:53.280 INFO 10 --- [pool-overview-container-6,,3d186c146f39] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=417 2023-11-27 13:35:53.319 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:53.319 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:53.340 INFO 10 --- [pool-overview-container-6,,3d186c146f39] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=16781 2023-11-27 13:35:53.343 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2023-11-27 13:35:53.370 INFO 10 --- [pool-Check host status3,,24738553ce85] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:35:53.402 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20231118 2023-11-27 13:35:53.405 INFO 10 --- [pool-Check host status7,,77c9c57b6af2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2023-11-27 13:35:53.409 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:53.409 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:53.414 INFO 10 --- [pool-Check host status4,,075a09f98e62] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-11-27 13:35:53.415 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` drop partition (P20231118) 2023-11-27 13:35:53.426 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.120.0.1:8080 2023-11-27 13:35:53.443 INFO 10 --- [pool-ocp-schedules-17,,9be614453cef] c.o.o.s.a.c.aggregate.AggregatorFactory : createAggregator, channelId=100001, channelName=自定义脚本通道(待配置样例) 2023-11-27 13:35:53.443 INFO 10 --- [pool-ocp-schedules-17,,9be614453cef] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=100001 2023-11-27 13:35:53.443 INFO 10 --- [pool-ocp-schedules-17,,9be614453cef] c.o.o.s.a.u.f.VersionalProductFactory : refresh done, configCount=2, updatedCount=2, failedToUpdateCount=0 2023-11-27 13:35:53.449 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.120.0.1:8080 2023-11-27 13:35:53.467 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:35:53.487 INFO 10 --- [pool-ocp-schedules-1,,e70cd7c90cd4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:35:53.510 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.120.0.1:8080 2023-11-27 13:35:53.558 INFO 10 --- [pool-ocp-schedules-17,,9be614453cef] c.o.o.s.a.core.process.AlarmProcessor : AlarmProcessor init done, lastProcessTime=2023-11-27T13:35:53.526441+08:00 2023-11-27 13:35:53.562 INFO 10 --- [pool-ocp-schedules-17,,9be614453cef] c.o.o.s.a.core.task.AlarmProcessTask : AlarmProcessTask created 2023-11-27 13:35:53.590 INFO 10 --- [pool-ocp-schedules-17,,9be614453cef] c.o.o.s.a.utils.task.AlarmTaskManager : start task, taskInfo=TaskInfo(id=2, uuid=15405f18-b190-4bf2-b8fc-c8aebe3ea16c, taskType=AlarmProcess, target=null, parameters=null, intervalSeconds=10, status=EXECUTING, executorName=10.120.0.1:10, createAt=2022-11-29T06:49:27.131956Z, startAt=2023-11-27T05:35:52.113Z, endAt=null, heartbeatAt=2023-11-27T05:35:52.113Z, result=executing) 2023-11-27 13:35:53.590 INFO 10 --- [pool-ocp-schedules-17,,9be614453cef] c.o.o.s.a.utils.task.AlarmTaskManager : startNeedExecute, activeCount=0, tryStartCount=2, startedCount=2 2023-11-27 13:35:53.605 INFO 10 --- [pool-ocp-schedules-9,,8ca48a74dead] c.o.o.s.a.c.aggregate.AggregatorFactory : createAggregator, channelId=100001, channelName=自定义脚本通道(待配置样例) 2023-11-27 13:35:53.605 INFO 10 --- [pool-ocp-schedules-9,,8ca48a74dead] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=100001 2023-11-27 13:35:53.605 INFO 10 --- [pool-ocp-schedules-9,,8ca48a74dead] c.o.o.s.a.u.f.VersionalProductFactory : refresh done, configCount=2, updatedCount=2, failedToUpdateCount=0 2023-11-27 13:35:53.616 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:35:53.706 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=obcluster, obClusterId=1, tenantName=sys, username=ocp_monitor, address=133.197.206.97, port=2881, database=oceanbase) 2023-11-27 13:35:53.706 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=direct, address=null, port=null, obsAddrList=[ObServerAddr(address=133.197.206.97, port=2881)], username=ocp_monitor, tenantName=sys, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2023-11-27 13:35:53.796 INFO 10 --- [http-nio-0.0.0.0-8080-exec-2,c299ce270b0e447b,acf8ead470b6] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.11, traceId=c299ce270b0e447b, duration=2214 ms] 2023-11-27 13:35:53.796 INFO 10 --- [http-nio-0.0.0.0-8080-exec-1,791a900eff1e4b4d,b9cf6ce9844d] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.12, traceId=791a900eff1e4b4d, duration=2214 ms] 2023-11-27 13:35:53.803 INFO 10 --- [obsdk-init-thread-2,,9c590ae2b6e7] com.alibaba.druid.pool.DruidDataSource : {dataSource-5,ob-connector-datasource_direct_sys_ocp_monitor_133.197.206.97_2881} inited 2023-11-27 13:35:53.803 INFO 10 --- [obsdk-init-thread-2,,9c590ae2b6e7] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource success, poolingCount=1, connectProperties=ConnectProperties(connectionMode=direct, address=133.197.206.97, port=2881, obsAddrList=[ObServerAddr(address=133.197.206.97, port=2881)], username=ocp_monitor, tenantName=sys, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2023-11-27 13:35:53.804 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector success 2023-11-27 13:35:53.804 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:35:53.826 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:35:53.826 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:53.828 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:35:53.839 INFO 10 --- [ForkJoinPool-5-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:35:53.840 INFO 10 --- [pool-ocp-schedules-17,,9be614453cef] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:35:53.840 INFO 10 --- [pool-ocp-schedules-17,,9be614453cef] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:35:53.918 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:53.918 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:53.921 INFO 10 --- [ForkJoinPool-5-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:35:53.921 INFO 10 --- [ForkJoinPool-5-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:53.924 INFO 10 --- [ForkJoinPool-5-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:35:53.949 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2023-11-27 13:35:53.971 INFO 10 --- [ForkJoinPool-5-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:35:53.971 INFO 10 --- [ForkJoinPool-5-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:53.984 INFO 10 --- [ForkJoinPool-5-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:35:53.989 INFO 10 --- [ForkJoinPool-5-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=3.1.4 2023-11-27 13:35:53.998 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:53.998 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:54.013 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:35:54.021 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:35:54.021 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:54.032 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, MAX(CASE `name` WHEN 'region' THEN `info` ELSE '' END ) `region`, MAX(CASE `name` WHEN 'idc' THEN `info` ELSE '' END ) `idc`, MAX(CASE `name` WHEN 'status' THEN `info` ELSE '' END ) `status`, MAX(CASE `name` WHEN 'merge_status' THEN `info` ELSE '' END ) `merge_status`, MAX(CASE `name` WHEN 'broadcast_version' THEN `value` ELSE 0 END ) `broadcast_version`, MAX(CASE `name` WHEN 'all_merged_version' THEN `value` ELSE 0 END ) `all_merged_version`, MAX(CASE `name` WHEN 'last_merged_version' THEN `value` ELSE 0 END ) `last_merged_version`, MAX(CASE `name` WHEN 'merge_start_time' THEN `value` ELSE 0 END ) `merge_start_time`, MAX(CASE `name` WHEN 'last_merged_time' THEN `value` ELSE 0 END ) `last_merged_time`, MAX(CASE `name` WHEN 'is_merge_timeout' THEN `value` ELSE 0 END ) `merge_timeout` FROM oceanbase.__all_zone WHERE `zone` <> '' GROUP BY `zone` 2023-11-27 13:35:54.045 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:54.045 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:54.102 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2023-11-27 13:35:54.119 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20231203 2023-11-27 13:35:54.128 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:54.128 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:54.133 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20231203 values less than (1701648000000000)) 2023-11-27 13:35:54.156 INFO 10 --- [http-nio-0.0.0.0-8080-exec-3,0b711acc5f5547ee,32dd7c62f4f5] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.12, traceId=0b711acc5f5547ee, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:35:54.234 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:35:54.480 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:35:54.635 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:35:54.658 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:54.658 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:54.679 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2023-11-27 13:35:54.862 INFO 10 --- [http-nio-0.0.0.0-8080-exec-3,0b711acc5f5547ee,32dd7c62f4f5] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.12, traceId=0b711acc5f5547ee, duration=715 ms] 2023-11-27 13:35:54.916 INFO 10 --- [SimpleAsyncTaskExecutor-11,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=proxy, clusterName=darenOB, obClusterId=10, tenantName=sys, username=ocp_monitor, address=127.0.0.1, port=2888, database=oceanbase) 2023-11-27 13:35:54.916 INFO 10 --- [SimpleAsyncTaskExecutor-11,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=ocp_monitor, tenantName=sys, clusterName=darenOB, obClusterId=10, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://133.197.206.97:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2023-11-27 13:35:54.979 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2023-11-27 13:35:55.014 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20231204 2023-11-27 13:35:55.029 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:55.029 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:55.036 INFO 10 --- [pool-ocp-schedules-1,,e70cd7c90cd4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:35:55.037 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20231204 values less than (1701734400000000)) 2023-11-27 13:35:55.104 INFO 10 --- [pool-ocp-schedules-11,,6d1f3280bb58] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, newSeriesIdsCount=20000, maxSyncedSeriesId=470534528970129408 2023-11-27 13:35:55.329 INFO 10 --- [pool-ocp-schedules-1,,e70cd7c90cd4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:35:55.396 INFO 10 --- [pool-Check host status8,,904533283ee8] c.o.o.e.internal.connector.Connectors : init global executor service. 2023-11-27 13:35:55.397 INFO 10 --- [pool-Check host status8,,904533283ee8] c.o.o.e.internal.connector.Connectors : init global connector holder. 2023-11-27 13:35:55.397 INFO 10 --- [pool-Check host status8,,904533283ee8] c.o.o.e.i.connector.ConnectorHolder : Init connector holder, maxIdleSecs=3600 2023-11-27 13:35:55.408 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:35:55.407106+08:00, lastProcessTime=2023-11-27T13:35:53.526441+08:00 2023-11-27 13:35:55.437 INFO 10 --- [pool-scanner--12-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=44 2023-11-27 13:35:55.519 INFO 10 --- [pool-ocp-schedules-1,,e70cd7c90cd4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-11-27 13:35:55.648 INFO 10 --- [pool-scanner--13-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=44 2023-11-27 13:35:55.729 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:55.729 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:55.742 INFO 10 --- [pool-ocp-schedules-1,,e70cd7c90cd4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-11-27 13:35:55.789 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2023-11-27 13:35:55.822 INFO 10 --- [SimpleAsyncTaskExecutor-9,,] c.o.o.c.t.e.coordinator.TaskCoordinator : Start task coordinator 2023-11-27 13:35:55.849 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2023-11-27 13:35:55.879 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20231205 2023-11-27 13:35:55.888 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:55.888 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:55.902 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20231205 values less than (1701820800000000)) 2023-11-27 13:35:55.920 INFO 10 --- [pool-ocp-schedules-1,,e70cd7c90cd4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2023-11-27 13:35:56.064 INFO 10 --- [pool-ocp-schedules-1,,e70cd7c90cd4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2023-11-27 13:35:56.146 INFO 10 --- [pool-ocp-schedules-1,,e70cd7c90cd4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:35:56.540 INFO 10 --- [http-nio-0.0.0.0-8080-exec-5,8540a50162f148d1,7365ffa8ccbf] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=ObIDCRegionInfo&User_ID=admin&UID=alibaba&ObRegion=darenOB&version=2&ObRegionId=10, client=133.197.206.97, traceId=8540a50162f148d1, method=OcpApiRtnObj com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObIDCRegionInfo(String,Long,String,Long,Integer), args=darenOB,10,2,] 2023-11-27 13:35:56.585 INFO 10 --- [pool-ocp-schedules-1,,e70cd7c90cd4] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, count=8 2023-11-27 13:35:56.681 INFO 10 --- [http-nio-0.0.0.0-8080-exec-5,8540a50162f148d1,7365ffa8ccbf] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.206.97, traceId=8540a50162f148d1, duration=145 ms] 2023-11-27 13:35:56.775 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:56.776 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:56.777 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2023-11-27 13:35:56.959 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2023-11-27 13:35:56.982 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20231116 2023-11-27 13:35:56.990 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:56.990 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:57.000 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` drop partition (P20231116) 2023-11-27 13:35:57.131 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:57.131 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:57.132 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2023-11-27 13:35:57.193 INFO 10 --- [obsdk-init-thread-3,,30de8e7ea491] com.alibaba.druid.pool.DruidDataSource : {dataSource-6,ob-connector-datasource_proxy_darenOB_10_sys_ocp_monitor_127.0.0.1_2888} inited 2023-11-27 13:35:57.195 INFO 10 --- [obsdk-init-thread-3,,30de8e7ea491] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource success, poolingCount=1, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=ocp_monitor, tenantName=sys, clusterName=darenOB, obClusterId=10, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://133.197.206.97:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2023-11-27 13:35:57.196 INFO 10 --- [SimpleAsyncTaskExecutor-11,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector success 2023-11-27 13:35:57.196 INFO 10 --- [SimpleAsyncTaskExecutor-11,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:35:57.206 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20231117 2023-11-27 13:35:57.226 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:57.226 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:57.237 INFO 10 --- [SimpleAsyncTaskExecutor-11,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:35:57.237 INFO 10 --- [SimpleAsyncTaskExecutor-11,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:57.252 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` drop partition (P20231117) 2023-11-27 13:35:57.252 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:35:57.253 INFO 10 --- [SimpleAsyncTaskExecutor-11,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:35:57.257 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:35:57.257 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:57.406 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:35:57.416 INFO 10 --- [ForkJoinPool-5-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:35:57.439 INFO 10 --- [SimpleAsyncTaskExecutor-11,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:35:57.439 INFO 10 --- [SimpleAsyncTaskExecutor-11,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:57.452 INFO 10 --- [SimpleAsyncTaskExecutor-11,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select event_id, wait_class as wait_class_name, wait_class_id, event as event_name from v$session_event where sid = (select sid from v$session_event limit 1) 2023-11-27 13:35:57.454 INFO 10 --- [ForkJoinPool-5-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:35:57.454 INFO 10 --- [ForkJoinPool-5-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:57.461 INFO 10 --- [ForkJoinPool-5-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:35:57.468 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:35:57.468 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:57.509 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-11-27 13:35:57.519 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:35:57.521 INFO 10 --- [ForkJoinPool-5-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:35:57.521 INFO 10 --- [ForkJoinPool-5-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:57.523 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:57.523 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:57.528 INFO 10 --- [ForkJoinPool-5-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:35:57.528 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:35:57.528 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2023-11-27 13:35:57.537 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:35:57.537 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:57.550 INFO 10 --- [ForkJoinPool-5-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-11-27 13:35:57.556 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:35:57.560 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:35:57.560 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:57.562 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, MAX(CASE `name` WHEN 'region' THEN `info` ELSE '' END ) `region`, MAX(CASE `name` WHEN 'idc' THEN `info` ELSE '' END ) `idc`, MAX(CASE `name` WHEN 'status' THEN `info` ELSE '' END ) `status`, MAX(CASE `name` WHEN 'merge_status' THEN `info` ELSE '' END ) `merge_status`, MAX(CASE `name` WHEN 'broadcast_version' THEN `value` ELSE 0 END ) `broadcast_version`, MAX(CASE `name` WHEN 'all_merged_version' THEN `value` ELSE 0 END ) `all_merged_version`, MAX(CASE `name` WHEN 'last_merged_version' THEN `value` ELSE 0 END ) `last_merged_version`, MAX(CASE `name` WHEN 'merge_start_time' THEN `value` ELSE 0 END ) `merge_start_time`, MAX(CASE `name` WHEN 'last_merged_time' THEN `value` ELSE 0 END ) `last_merged_time`, MAX(CASE `name` WHEN 'is_merge_timeout' THEN `value` ELSE 0 END ) `merge_timeout` FROM oceanbase.__all_zone WHERE `zone` <> '' GROUP BY `zone` 2023-11-27 13:35:57.565 WARN 10 --- [ForkJoinPool-6-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-11-27 13:35:57.583 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20231118 2023-11-27 13:35:57.593 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:57.593 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:57.594 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:35:57.594 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:57.597 INFO 10 --- [SimpleAsyncTaskExecutor-9,,] c.o.ocp.server.config.TaskEngineRunner : Startup subtask coordinator. subtask=403 2023-11-27 13:35:57.597 INFO 10 --- [SimpleAsyncTaskExecutor-9,,] c.o.o.c.t.e.c.SubtaskCoordinator : Start subtask coordinator 2023-11-27 13:35:57.602 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, stop_time, start_service_time, block_migrate_in_time, id FROM oceanbase.__all_server ORDER BY zone ASC, id ASC 2023-11-27 13:35:57.610 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` drop partition (P20231118) 2023-11-27 13:35:57.613 INFO 10 --- [ForkJoinPool-6-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.206.97:2882, status=ACTIVE, startServiceTime=1701063018577428, active=true 2023-11-27 13:35:57.613 INFO 10 --- [ForkJoinPool-6-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=133.197.206.97:2882, stopTime=0, stopped=false 2023-11-27 13:35:57.616 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=IDLE, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=IDLE, startTime=null, stopTime=null} 2023-11-27 13:35:57.617 INFO 10 --- [pool-ocp-schedules-11,,6d1f3280bb58] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, newSeriesIdsCount=20000, maxSyncedSeriesId=494705779422658560 2023-11-27 13:35:57.622 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.d.MessageDistributorFactory : build alarmSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2023-11-27T13:35:26.395+08:00_groups-8-2023-04-25T06:45:26Z_templateDetectRules-127-2023-04-25T06:45:29.278901Z_readableClusters_2, subscribedJudgerVersion 2023-11-27 13:35:57.639 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:35:57.643 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=16781 2023-11-27 13:35:57.653 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:35:57.663 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.d.DistributeBuilder : buildSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2023-11-27 13:35:57.663 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.d.MessageDistributorFactory : build recoverSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2023-11-27T13:35:26.395+08:00_groups-8-2023-04-25T06:45:26Z_templateDetectRules-127-2023-04-25T06:45:29.278901Z_readableClusters_2, subscribedJudgerVersion 2023-11-27 13:35:57.663 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.d.DistributeBuilder : buildRecoverMessageSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2023-11-27 13:35:57.664 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.d.MessageDistributorFactory : buildAllDistributors 2023-11-27 13:35:57.701 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605111] update cluster status, clusterId=2, oldStatus=UNAVAILABLE, newStatus=RUNNING 2023-11-27 13:35:57.710 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:35:55.407106+08:00 2023-11-27 13:35:57.863 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=IDLE, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:35:57.867 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=133.197.206.97, port=2882, sqlPort=2881, version=3.1.4_10000092022071511-b4bfa011ceaef428782dcb65ae89190c40b78c2f(Jul 15 2022 11:45:14), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2022-11-29T14:36:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.206.97, port=2882, sqlPort=2881, version=3.1.4_10000092022071511-b4bfa011ceaef428782dcb65ae89190c40b78c2f(Jul 15 2022 11:45:14), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-27T05:30:18.577428Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:35:57.914 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=16781 2023-11-27 13:35:58.019 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:58.019 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:58.027 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2023-11-27 13:35:58.153 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=133.197.206.97, port=2882, sqlPort=2881, version=3.1.4_10000092022071511-b4bfa011ceaef428782dcb65ae89190c40b78c2f(Jul 15 2022 11:45:14), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-27T05:30:18.577428Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:35:58.175 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:58.175 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:58.178 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:35:58.203 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:58.203 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:58.224 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2023-11-27 13:35:58.260 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20231203 2023-11-27 13:35:58.277 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:58.277 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:58.300 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20231203 values less than (1701648000000000)) 2023-11-27 13:35:58.336 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:35:58.346 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:35:58.349 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:35:58.349 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:58.354 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:35:58.364 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:35:58.365 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:58.374 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:35:58.619 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:35:58.621 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:35:58.622 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:35:58.622 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:58.622 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:35:58.624 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:35:58.624 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:58.625 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:35:58.640 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:35:58.641 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:58.644 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT cluster_id, cluster_name, created, cluster_role, cluster_status, current_scn, standby_became_primary_scn, primary_cluster_id, (time_to_usec(now(6)) - current_scn)/1000000 as delay_time, protection_mode, redo_transport_options, protection_level FROM oceanbase.v$ob_cluster 2023-11-27 13:35:58.658 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=16781 2023-11-27 13:35:58.728 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:35:58.728 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:58.729 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT switchover_status, switchover_info FROM oceanbase.v$ob_cluster 2023-11-27 13:35:58.747 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:58.747 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:58.769 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2023-11-27 13:35:58.773 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=16781 2023-11-27 13:35:58.841 INFO 10 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2023-11-27 13:35:58.848 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=obcluster, obClusterId=1 2023-11-27 13:35:58.848 INFO 10 --- [ForkJoinPool-6-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"133.197.206.97:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1700793672036000}, createTime=2022-11-29T14:52:53+08:00, creator=admin, obVersion=3.1.4, tenantCount=3) 2023-11-27 13:35:58.877 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2023-11-27 13:35:58.922 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20231204 2023-11-27 13:35:58.949 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:58.950 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:58.976 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20231204 values less than (1701734400000000)) 2023-11-27 13:35:59.044 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:35:59.054 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:35:59.054 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:59.061 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=16781 2023-11-27 13:35:59.077 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:35:59.103 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:35:59.103 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:59.120 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, stop_time, start_service_time, block_migrate_in_time, id FROM oceanbase.__all_server ORDER BY zone ASC, id ASC 2023-11-27 13:35:59.215 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=16781 2023-11-27 13:35:59.435 INFO 10 --- [pool-ocp-schedules-11,,6d1f3280bb58] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, newSeriesIdsCount=20000, maxSyncedSeriesId=507369707030052864 2023-11-27 13:35:59.464 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:35:59.472 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:35:59.508 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:59.517 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:35:59.547 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:35:59.547 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:59.548 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-11-27 13:35:59.578 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:59.578 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:59.581 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2023-11-27 13:35:59.583 INFO 10 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (darenOB:10-3) 2023-11-27 13:35:59.586 INFO 10 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-1) 2023-11-27 13:35:59.594 INFO 10 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (darenOB:10-4) 2023-11-27 13:35:59.619 INFO 10 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (darenOB:10-1) 2023-11-27 13:35:59.633 INFO 10 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (darenOB:10-5) 2023-11-27 13:35:59.645 INFO 10 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (darenOB:10-6) 2023-11-27 13:35:59.760 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2023-11-27 13:35:59.793 INFO 10 --- [http-nio-0.0.0.0-8080-exec-7,42d784b207be4d0e,f34e903867be] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=133.197.206.97, traceId=42d784b207be4d0e, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-11-27 13:35:59.796 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20231205 2023-11-27 13:35:59.802 INFO 10 --- [http-nio-0.0.0.0-8080-exec-7,42d784b207be4d0e,f34e903867be] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.206.97, traceId=42d784b207be4d0e, duration=10 ms] 2023-11-27 13:35:59.854 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:35:59.854 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:35:59.873 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20231205 values less than (1701820800000000)) 2023-11-27 13:35:59.972 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:35:59.973 INFO 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=1, level=0->1, c=0) - not enough cached interval (source=2023-04-25T14:52:31, target=2023-04-25T14:50:31, granule=120000ms, delay=120000ms) 2023-11-27 13:35:59.973 INFO 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=1, level=1->2, c=0) - not enough cached interval (source=2023-04-25T14:50:31, target=2023-04-25T14:46:01, granule=600000ms, delay=270000ms) 2023-11-27 13:36:00.032 INFO 10 --- [pool-ocp-schedules-13,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-11-27 13:36:00.032 INFO 10 --- [pool-ocp-schedules-13,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1701063360 2023-11-27 13:36:00.032 INFO 10 --- [pool-ocp-schedules-3,,] c.o.o.s.task.scheduled.OcpSchedules : Refresh OCP properties. 2023-11-27 13:36:00.033 INFO 10 --- [pool-ocp-schedules-3,,] c.o.ocp.core.property.PropertyService : Refresh config properties 2023-11-27 13:36:00.035 INFO 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=darenOB:10, server=1, level=0->1, c=0) - not enough cached interval (source=2023-11-24T10:14:01, target=2023-11-24T10:11:53, granule=120000ms, delay=128000ms) 2023-11-27 13:36:00.035 INFO 10 --- [pool-ocp-schedules-8,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-11-27 13:36:00.035 INFO 10 --- [pool-ocp-schedules-8,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-11-27 13:36:00.035 INFO 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=darenOB:10, server=1, level=1->2, c=0) - not enough cached interval (source=2023-11-24T10:11:53, target=2023-11-24T10:05:53, granule=600000ms, delay=360000ms) 2023-11-27 13:36:00.084 INFO 10 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:00.084 INFO 10 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:00.098 INFO 10 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:00.116 INFO 10 --- [sql-rollup-worker-4,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:00.116 INFO 10 --- [sql-rollup-worker-4,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:00.134 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=darenOB:10, server=3, level=0->1, c=0) - fall behind, will continue from 2023-11-26T13:36:00 (source=2023-11-27T13:33:32, target=2023-11-24T10:19:23, granule=120000ms, delay=270849000ms) 2023-11-27 13:36:00.166 INFO 10 --- [sql-rollup-worker-4,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:00.200 INFO 10 --- [sql-rollup-worker-5,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:00.200 INFO 10 --- [sql-rollup-worker-5,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:00.204 INFO 10 --- [pool-ocp-schedules-8,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-11-27 13:36:00.206 INFO 10 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : new: [] 2023-11-27 13:36:00.206 INFO 10 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : init spring 2023-11-27 13:36:00.206 INFO 10 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2023-11-27 13:36:00.206 INFO 10 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : starting 2023-11-27 13:36:00.212 INFO 10 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : new: [] 2023-11-27 13:36:00.212 INFO 10 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : init spring 2023-11-27 13:36:00.212 INFO 10 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2023-11-27 13:36:00.212 INFO 10 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : starting 2023-11-27 13:36:00.263 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=darenOB:10, server=5, level=0->1, c=0) - fall behind, will continue from 2023-11-26T13:36:00 (source=2023-11-27T13:33:32, target=2023-11-24T10:19:23, granule=120000ms, delay=270849000ms) 2023-11-27 13:36:00.265 INFO 10 --- [sql-rollup-worker-5,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:00.266 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=darenOB:10, server=6, level=0->1, c=0) - fall behind, will continue from 2023-11-26T13:36:00 (source=2023-11-27T13:33:32, target=2023-11-24T10:19:23, granule=120000ms, delay=270849000ms) 2023-11-27 13:36:00.294 INFO 10 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:00.294 INFO 10 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:00.314 INFO 10 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:00.314 INFO 10 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:00.320 INFO 10 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:00.320 INFO 10 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:00.321 INFO 10 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2023-11-27 13:36:00.330 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=darenOB:10, server=4, level=0->1, c=0) - fall behind, will continue from 2023-11-26T13:36:00 (source=2023-11-27T13:33:31, target=2023-11-24T10:19:23, granule=120000ms, delay=270848000ms) 2023-11-27 13:36:00.344 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:00.344 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:00.352 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2023-11-27 13:36:00.545 INFO 10 --- [pool-Check host status6,,144f4493b486] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=http, hostAddress=133.197.204.12, httpPort=62888, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=basic, basicAuthConfig=BasicAuthConfig(username=ocp_agent), digestAuthConfig=null, formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), posConfig=null, proxy=null), estimated size:1 2023-11-27 13:36:00.546 INFO 10 --- [pool-Check host status6,,144f4493b486] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.12:62888/api/v1/time, params:null 2023-11-27 13:36:00.547 INFO 10 --- [pool-Check host status1,,16cff7425ff1] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=http, hostAddress=133.197.206.97, httpPort=62888, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=basic, basicAuthConfig=BasicAuthConfig(username=ocp_agent), digestAuthConfig=null, formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), posConfig=null, proxy=null), estimated size:4 2023-11-27 13:36:00.547 INFO 10 --- [pool-Check host status1,,16cff7425ff1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.97:62888/api/v1/time, params:null 2023-11-27 13:36:00.548 INFO 10 --- [pool-Check host status7,,77c9c57b6af2] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=http, hostAddress=133.197.204.13, httpPort=62888, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=basic, basicAuthConfig=BasicAuthConfig(username=ocp_agent), digestAuthConfig=null, formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), posConfig=null, proxy=null), estimated size:5 2023-11-27 13:36:00.548 INFO 10 --- [pool-Check host status7,,77c9c57b6af2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.13:62888/api/v1/time, params:null 2023-11-27 13:36:00.549 INFO 10 --- [pool-Check host status2,,daf979afdf7e] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=http, hostAddress=133.197.204.3, httpPort=62888, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=basic, basicAuthConfig=BasicAuthConfig(username=ocp_agent), digestAuthConfig=null, formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), posConfig=null, proxy=null), estimated size:6 2023-11-27 13:36:00.549 INFO 10 --- [pool-Check host status2,,daf979afdf7e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.3:62888/api/v1/time, params:null 2023-11-27 13:36:00.549 INFO 10 --- [pool-Check host status8,,904533283ee8] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=http, hostAddress=133.197.206.88, httpPort=62888, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=basic, basicAuthConfig=BasicAuthConfig(username=ocp_agent), digestAuthConfig=null, formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), posConfig=null, proxy=null), estimated size:7 2023-11-27 13:36:00.549 INFO 10 --- [pool-Check host status8,,904533283ee8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.88:62888/api/v1/time, params:null 2023-11-27 13:36:00.545 INFO 10 --- [pool-Check host status4,,075a09f98e62] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=http, hostAddress=133.197.204.10, httpPort=62888, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=basic, basicAuthConfig=BasicAuthConfig(username=ocp_agent), digestAuthConfig=null, formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), posConfig=null, proxy=null), estimated size:2 2023-11-27 13:36:00.549 INFO 10 --- [pool-Check host status4,,075a09f98e62] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.10:62888/api/v1/time, params:null 2023-11-27 13:36:00.545 INFO 10 --- [pool-Check host status5,,ee500bda988b] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=http, hostAddress=133.197.204.11, httpPort=62888, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=basic, basicAuthConfig=BasicAuthConfig(username=ocp_agent), digestAuthConfig=null, formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), posConfig=null, proxy=null), estimated size:3 2023-11-27 13:36:00.550 INFO 10 --- [pool-Check host status5,,ee500bda988b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.11:62888/api/v1/time, params:null 2023-11-27 13:36:00.550 INFO 10 --- [pool-Check host status3,,24738553ce85] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=http, hostAddress=133.197.206.92, httpPort=62888, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=basic, basicAuthConfig=BasicAuthConfig(username=ocp_agent), digestAuthConfig=null, formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), posConfig=null, proxy=null), estimated size:8 2023-11-27 13:36:00.550 INFO 10 --- [pool-Check host status3,,24738553ce85] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.92:62888/api/v1/time, params:null 2023-11-27 13:36:00.592 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2023-11-27 13:36:00.614 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=0) - no updates (start=2023-11-26T13:36:00, end=2023-11-26T13:38:30) 2023-11-27 13:36:00.623 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:00.624 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:00.626 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:00.626 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:00.634 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:00.637 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20231116 2023-11-27 13:36:00.645 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:00.645 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:00.653 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=0) - no updates (start=2023-11-26T13:36:00, end=2023-11-26T13:38:30) 2023-11-27 13:36:00.653 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=0) - no updates (start=2023-11-26T13:36:00, end=2023-11-26T13:38:30) 2023-11-27 13:36:00.653 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:36:00.656 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=0) - no updates (start=2023-11-26T13:36:00, end=2023-11-26T13:38:30) 2023-11-27 13:36:00.657 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-11-27 13:36:00.665 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:00.665 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:00.667 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:36:00.675 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` drop partition (P20231116) 2023-11-27 13:36:00.676 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:00.676 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:00.690 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-11-27 13:36:00.695 INFO 10 --- [ForkJoinPool-6-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.204.3:2882, status=INACTIVE, startServiceTime=0, active=false 2023-11-27 13:36:00.695 INFO 10 --- [ForkJoinPool-6-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.204.10:2882, status=ACTIVE, startServiceTime=1695730092226557, active=true 2023-11-27 13:36:00.695 INFO 10 --- [ForkJoinPool-6-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=133.197.204.10:2882, stopTime=0, stopped=false 2023-11-27 13:36:00.695 INFO 10 --- [ForkJoinPool-6-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.204.11:2882, status=ACTIVE, startServiceTime=1695730305487530, active=true 2023-11-27 13:36:00.695 INFO 10 --- [ForkJoinPool-6-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=133.197.204.11:2882, stopTime=0, stopped=false 2023-11-27 13:36:00.695 INFO 10 --- [ForkJoinPool-6-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.204.12:2882, status=ACTIVE, startServiceTime=1695730736949667, active=true 2023-11-27 13:36:00.695 INFO 10 --- [ForkJoinPool-6-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=133.197.204.12:2882, stopTime=0, stopped=false 2023-11-27 13:36:00.695 INFO 10 --- [ForkJoinPool-6-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.204.13:2882, status=ACTIVE, startServiceTime=1695731005656813, active=true 2023-11-27 13:36:00.695 INFO 10 --- [ForkJoinPool-6-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=133.197.204.13:2882, stopTime=0, stopped=false 2023-11-27 13:36:00.695 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=14, name=zone6, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone6, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-27 13:36:00.714 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=1) - no updates (start=2023-11-26T13:38:30, end=2023-11-26T13:41:00) 2023-11-27 13:36:00.732 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=1) - no updates (start=2023-11-26T13:38:30, end=2023-11-26T13:41:00) 2023-11-27 13:36:00.752 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=1) - no updates (start=2023-11-26T13:38:30, end=2023-11-26T13:41:00) 2023-11-27 13:36:00.756 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=1) - no updates (start=2023-11-26T13:38:30, end=2023-11-26T13:41:00) 2023-11-27 13:36:00.762 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_timeout, expression merge_timeout_flag == 1 2023-11-27 13:36:00.769 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=2) - no updates (start=2023-11-26T13:41:00, end=2023-11-26T13:43:30) 2023-11-27 13:36:00.790 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_timeout_flag', labels={}}] 2023-11-27 13:36:00.791 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_timeout 2023-11-27 13:36:00.791 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_timeout 2023-11-27 13:36:00.792 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_drop, expression net_receive_drop_count > 1 2023-11-27 13:36:00.794 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_receive_drop_count', labels={}}] 2023-11-27 13:36:00.794 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_drop 2023-11-27 13:36:00.794 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_drop 2023-11-27 13:36:00.794 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_host_task_timeout, expression ob_server_task_max_duration_seconds > 10800 2023-11-27 13:36:00.795 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_task_max_duration', labels={}}] 2023-11-27 13:36:00.796 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_task_timeout 2023-11-27 13:36:00.796 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_host_task_timeout 2023-11-27 13:36:00.796 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_cpu_percent_over_threshold, expression ob_cpu_percent > 85 2023-11-27 13:36:00.796 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cpu_percent', labels={}}] 2023-11-27 13:36:00.796 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cpu_percent_over_threshold 2023-11-27 13:36:00.796 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_cpu_percent_over_threshold 2023-11-27 13:36:00.796 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_sys_abnormal, expression cpu_system > 40 2023-11-27 13:36:00.796 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_system', labels={}}] 2023-11-27 13:36:00.796 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_sys_abnormal 2023-11-27 13:36:00.796 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_sys_abnormal 2023-11-27 13:36:00.796 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ocp_alarm_detect_timeout, expression ocp_alarm_detect_duration_seconds >= 2.5 2023-11-27 13:36:00.797 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_alarm_detect_duration_seconds', labels={}}] 2023-11-27 13:36:00.797 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_alarm_detect_timeout 2023-11-27 13:36:00.797 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ocp_alarm_detect_timeout 2023-11-27 13:36:00.797 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_cluster_status_check_failed, expression ob_cluster_status_check == 0 2023-11-27 13:36:00.797 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_status_check', labels={}}] 2023-11-27 13:36:00.797 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_status_check_failed 2023-11-27 13:36:00.797 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_status_check_failed 2023-11-27 13:36:00.798 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_retry_times_exceeded, expression storage_capacity_monitor_error_code == 1001 2023-11-27 13:36:00.798 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}] 2023-11-27 13:36:00.798 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_retry_times_exceeded 2023-11-27 13:36:00.798 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_retry_times_exceeded 2023-11-27 13:36:00.798 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 95 2023-11-27 13:36:00.798 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 92 2023-11-27 13:36:00.798 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_log_path_disk_percent', labels={}}] 2023-11-27 13:36:00.798 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_log_disk_percent_over_threshold 2023-11-27 13:36:00.798 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_host_log_disk_percent_over_threshold 2023-11-27 13:36:00.799 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_host_tcp_retrans_percent_over_threshold, expression tcp_retrans_percent > 10 2023-11-27 13:36:00.799 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_tcp_retrans_percent', labels={}}] 2023-11-27 13:36:00.799 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_tcp_retrans_percent_over_threshold 2023-11-27 13:36:00.799 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_host_tcp_retrans_percent_over_threshold 2023-11-27 13:36:00.799 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_host_cpu_percent_over_threshold, expression cpu_percent > 90 2023-11-27 13:36:00.799 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_cpu_percent', labels={}}] 2023-11-27 13:36:00.799 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_cpu_percent_over_threshold 2023-11-27 13:36:00.803 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_host_cpu_percent_over_threshold 2023-11-27 13:36:00.803 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for obproxy_cannot_connected, expression obproxy_connectable == 0 2023-11-27 13:36:00.835 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=2) - no updates (start=2023-11-26T13:41:00, end=2023-11-26T13:43:30) 2023-11-27 13:36:00.836 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_connectable', labels={}}] 2023-11-27 13:36:00.836 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_cannot_connected 2023-11-27 13:36:00.836 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-obproxy_cannot_connected 2023-11-27 13:36:00.836 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_hwm, expression cpu_used_percent > 90 and cpu_steal < 20 2023-11-27 13:36:00.837 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_steal', labels={}}, Metric{name='host_cpu_used_percent', labels={}}] 2023-11-27 13:36:00.837 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_hwm 2023-11-27 13:36:00.837 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_hwm 2023-11-27 13:36:00.837 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_hwm, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2023-11-27 13:36:00.837 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}] 2023-11-27 13:36:00.837 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_hwm 2023-11-27 13:36:00.837 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_hwm 2023-11-27 13:36:00.839 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 2 2023-11-27 13:36:00.839 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 1.5 2023-11-27 13:36:00.856 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_load1_per_cpu', labels={}}] 2023-11-27 13:36:00.856 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_load1_per_cpu_over_threshold 2023-11-27 13:36:00.856 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_host_load1_per_cpu_over_threshold 2023-11-27 13:36:00.856 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ocp_meta_db_disconnected, expression ocp_meta_db_health == 0 2023-11-27 13:36:00.856 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_meta_db_health', labels={}}] 2023-11-27 13:36:00.856 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_meta_db_disconnected 2023-11-27 13:36:00.856 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ocp_meta_db_disconnected 2023-11-27 13:36:00.856 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ocp_http_request_too_many_errors_occur, expression ocp_http_request_error_count >= 1 2023-11-27 13:36:00.856 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_error_count', labels={}}] 2023-11-27 13:36:00.857 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_too_many_errors_occur 2023-11-27 13:36:00.857 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_too_many_errors_occur 2023-11-27 13:36:00.857 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_tenant_compaction_error, expression compaction_is_error == 1 2023-11-27 13:36:00.858 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_compaction_error_flag', labels={}}] 2023-11-27 13:36:00.859 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_compaction_error 2023-11-27 13:36:00.859 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_compaction_error 2023-11-27 13:36:00.859 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for host_disk_readonly, expression disk_readonly_flag == 1 2023-11-27 13:36:00.859 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_disk_readonly_flag', labels={}}] 2023-11-27 13:36:00.859 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_disk_readonly 2023-11-27 13:36:00.859 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-host_disk_readonly 2023-11-27 13:36:00.859 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_delay_time_too_long, expression sync_delay_time > 600 2023-11-27 13:36:00.859 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync_delay_time', labels={}}] 2023-11-27 13:36:00.859 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_delay_time_too_long 2023-11-27 13:36:00.859 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_delay_time_too_long 2023-11-27 13:36:00.859 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ocp_http_request_timeout, expression ocp_http_request_duration_seconds >= 5 2023-11-27 13:36:00.860 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_duration_seconds', labels={}}] 2023-11-27 13:36:00.860 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_timeout 2023-11-27 13:36:00.860 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_timeout 2023-11-27 13:36:00.860 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_host_net_send_percent_over_threshold, expression net_send_percent > 80 2023-11-27 13:36:00.860 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_net_send_percent', labels={}}] 2023-11-27 13:36:00.860 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_send_percent_over_threshold 2023-11-27 13:36:00.860 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_send_percent_over_threshold 2023-11-27 13:36:00.860 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for tenant_cpu_percent_over_threshold, expression ob_cpu_percent > 95 2023-11-27 13:36:00.860 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_cpu_percent', labels={}}] 2023-11-27 13:36:00.860 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_cpu_percent_over_threshold 2023-11-27 13:36:00.860 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-tenant_cpu_percent_over_threshold 2023-11-27 13:36:00.860 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for os_tsar_sda_ioawait, expression io_await >= 200 and io_util >= 99 2023-11-27 13:36:00.861 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}] 2023-11-27 13:36:00.861 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_sda_ioawait 2023-11-27 13:36:00.861 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_sda_ioawait 2023-11-27 13:36:00.861 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for os_cpu_irq_error, expression cpu_softirq_per_cpu > 3 and cpu_idle_per_cpu < 6 2023-11-27 13:36:00.861 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_idle_per_cpu', labels={}}, Metric{name='host_cpu_softirq_per_cpu', labels={}}] 2023-11-27 13:36:00.861 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_cpu_irq_error 2023-11-27 13:36:00.861 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-os_cpu_irq_error 2023-11-27 13:36:00.861 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for os_tsar_nvme_ioawait, expression io_await > 20 and io_util >= 98 and io_qusize >= 20 2023-11-27 13:36:00.862 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}] 2023-11-27 13:36:00.862 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_nvme_ioawait 2023-11-27 13:36:00.862 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_nvme_ioawait 2023-11-27 13:36:00.862 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_host_data_disk_percent_over_threshold, expression ob_data_disk_usage_percent > 97 2023-11-27 13:36:00.862 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_data_path_disk_percent', labels={}}] 2023-11-27 13:36:00.862 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_data_disk_percent_over_threshold 2023-11-27 13:36:00.862 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_host_data_disk_percent_over_threshold 2023-11-27 13:36:00.862 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ic_server_connect_failed, expression ic_server_health == 0 2023-11-27 13:36:00.862 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ic_server_health', labels={}}] 2023-11-27 13:36:00.862 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ic_server_connect_failed 2023-11-27 13:36:00.862 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ic_server_connect_failed 2023-11-27 13:36:00.862 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_failed, expression ob_cluster_sync == 0 2023-11-27 13:36:00.863 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync', labels={}}] 2023-11-27 13:36:00.863 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_failed 2023-11-27 13:36:00.863 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_failed 2023-11-27 13:36:00.863 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_disk_usage_high, expression ob_tenant_log_disk_usage > 95 2023-11-27 13:36:00.863 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_disk_usage', labels={}}] 2023-11-27 13:36:00.863 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_disk_usage_high 2023-11-27 13:36:00.863 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_disk_usage_high 2023-11-27 13:36:00.863 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_error, expression net_traffic_err_count > 5 2023-11-27 13:36:00.863 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_traffic_err_count', labels={}}] 2023-11-27 13:36:00.863 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_error 2023-11-27 13:36:00.863 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_error 2023-11-27 13:36:00.863 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 95 2023-11-27 13:36:00.863 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 85 2023-11-27 13:36:00.864 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_memstore_percent', labels={}}] 2023-11-27 13:36:00.864 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_memstore_percent_over_threshold 2023-11-27 13:36:00.864 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-tenant_memstore_percent_over_threshold 2023-11-27 13:36:00.864 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for tenant_active_memstore_percent_over_threshold, expression active_memstore_percent > 110 2023-11-27 13:36:00.864 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_active_memstore_percent', labels={}}] 2023-11-27 13:36:00.864 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_active_memstore_percent_over_threshold 2023-11-27 13:36:00.864 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-tenant_active_memstore_percent_over_threshold 2023-11-27 13:36:00.864 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for obagent_dead, expression host_agent_process_status == 0 2023-11-27 13:36:00.866 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=2) - no updates (start=2023-11-26T13:41:00, end=2023-11-26T13:43:30) 2023-11-27 13:36:00.867 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=3) - no updates (start=2023-11-26T13:43:30, end=2023-11-26T13:46:00) 2023-11-27 13:36:00.869 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=2) - no updates (start=2023-11-26T13:41:00, end=2023-11-26T13:43:30) 2023-11-27 13:36:00.874 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=3) - no updates (start=2023-11-26T13:43:30, end=2023-11-26T13:46:00) 2023-11-27 13:36:00.882 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_process_status', labels={}}] 2023-11-27 13:36:00.882 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obagent_dead 2023-11-27 13:36:00.882 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-obagent_dead 2023-11-27 13:36:00.882 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_error, expression merge_error_flag == 1 2023-11-27 13:36:00.882 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_error_flag', labels={}}] 2023-11-27 13:36:00.882 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_error 2023-11-27 13:36:00.882 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_error 2023-11-27 13:36:00.882 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for os_observer_not_exist, expression observer_process_exists == 0 2023-11-27 13:36:00.883 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_process_exists', labels={}}] 2023-11-27 13:36:00.883 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_not_exist 2023-11-27 13:36:00.883 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-os_observer_not_exist 2023-11-27 13:36:00.883 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for host_agent_open_fd_count_over_threshold, expression host_agent_open_fd_count > 1000 2023-11-27 13:36:00.883 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_open_fd_count', labels={}}] 2023-11-27 13:36:00.883 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_open_fd_count_over_threshold 2023-11-27 13:36:00.883 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-host_agent_open_fd_count_over_threshold 2023-11-27 13:36:00.883 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_full, expression eth_net_traffic_usage > 92 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2023-11-27 13:36:00.883 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}] 2023-11-27 13:36:00.883 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_full 2023-11-27 13:36:00.883 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_full 2023-11-27 13:36:00.884 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_timeout_or_interrupted, expression storage_capacity_monitor_error_code == 1002 2023-11-27 13:36:00.885 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}] 2023-11-27 13:36:00.885 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_timeout_or_interrupted 2023-11-27 13:36:00.885 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_timeout_or_interrupted 2023-11-27 13:36:00.885 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_host_net_recv_percent_over_threshold, expression net_recv_percent > 80 2023-11-27 13:36:00.885 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_net_recv_percent', labels={}}] 2023-11-27 13:36:00.885 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_recv_percent_over_threshold 2023-11-27 13:36:00.885 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_recv_percent_over_threshold 2023-11-27 13:36:00.885 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for obproxyd_process_dead, expression obproxyd_process_exists == 0 2023-11-27 13:36:00.885 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_process_exists', labels={}}] 2023-11-27 13:36:00.885 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_dead 2023-11-27 13:36:00.893 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_dead 2023-11-27 13:36:00.893 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_zone_sstable_percent_over_threshold, expression ob_disk_percent > 97 2023-11-27 13:36:00.893 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_zone_sstable_percent_over_threshold, expression ob_disk_percent > 85 2023-11-27 13:36:00.893 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_zone_disk_percent', labels={}}] 2023-11-27 13:36:00.893 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_zone_sstable_percent_over_threshold 2023-11-27 13:36:00.893 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_zone_sstable_percent_over_threshold 2023-11-27 13:36:00.894 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 97 2023-11-27 13:36:00.894 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 95 2023-11-27 13:36:00.894 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_install_path_disk_percent', labels={}}] 2023-11-27 13:36:00.894 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_install_disk_percent_over_threshold 2023-11-27 13:36:00.894 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_host_install_disk_percent_over_threshold 2023-11-27 13:36:00.894 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_frozen, expression no_frozen_time > 90000 2023-11-27 13:36:00.894 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_frozen_seconds', labels={}}] 2023-11-27 13:36:00.894 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_frozen 2023-11-27 13:36:00.894 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_frozen 2023-11-27 13:36:00.894 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_cluster_frozen_version_delta_over_threshold, expression frozen_versions > 1 2023-11-27 13:36:00.894 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_frozen_version_delta', labels={}}] 2023-11-27 13:36:00.895 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_frozen_version_delta_over_threshold 2023-11-27 13:36:00.895 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_frozen_version_delta_over_threshold 2023-11-27 13:36:00.895 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_percent_over_threshold, expression tenant500_mem_hold_percent > 95 2023-11-27 13:36:00.895 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_percent', labels={}}] 2023-11-27 13:36:00.895 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_percent_over_threshold 2023-11-27 13:36:00.895 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_percent_over_threshold 2023-11-27 13:36:00.895 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for vpc_connect_failed, expression vpc_network_health == 0 2023-11-27 13:36:00.895 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='vpc_network_health', labels={}}] 2023-11-27 13:36:00.895 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=vpc_connect_failed 2023-11-27 13:36:00.895 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-vpc_connect_failed 2023-11-27 13:36:00.895 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for os_observer_fd_usage, expression observer_fd_usage > 65 and observer_fd_count > 0 2023-11-27 13:36:00.901 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_fd_count', labels={}}, Metric{name='ob_host_observer_fd_usage', labels={}}] 2023-11-27 13:36:00.901 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_fd_usage 2023-11-27 13:36:00.901 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-os_observer_fd_usage 2023-11-27 13:36:00.901 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for obproxy_process_dead, expression obproxy_process_exists == 0 2023-11-27 13:36:00.901 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_process_exists', labels={}}] 2023-11-27 13:36:00.901 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_dead 2023-11-27 13:36:00.901 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_dead 2023-11-27 13:36:00.901 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for node_file_root_usage, expression root_disk_usage > 95 2023-11-27 13:36:00.901 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_root_disk_usage', labels={}}] 2023-11-27 13:36:00.901 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_root_usage 2023-11-27 13:36:00.901 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-node_file_root_usage 2023-11-27 13:36:00.902 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_frozen, expression no_frozen_seconds > 90000 2023-11-27 13:36:00.902 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_frozen_seconds', labels={}}] 2023-11-27 13:36:00.902 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_frozen 2023-11-27 13:36:00.902 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_frozen 2023-11-27 13:36:00.902 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for os_observer_core_dump, expression observer_core_dump_seconds < 1200 and observer_core_dump_seconds > 0 2023-11-27 13:36:00.902 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_core_dump_seconds', labels={}}] 2023-11-27 13:36:00.902 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_core_dump 2023-11-27 13:36:00.902 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-os_observer_core_dump 2023-11-27 13:36:00.902 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 97 2023-11-27 13:36:00.902 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 85 2023-11-27 13:36:00.902 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_disk_percent', labels={}}] 2023-11-27 13:36:00.902 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_sstable_percent_over_threshold 2023-11-27 13:36:00.903 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_server_sstable_percent_over_threshold 2023-11-27 13:36:00.903 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_over_threshold, expression tenant500_mem_hold > 100 2023-11-27 13:36:00.903 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_gigabyte', labels={}}] 2023-11-27 13:36:00.903 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_over_threshold 2023-11-27 13:36:00.903 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_over_threshold 2023-11-27 13:36:00.903 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for os_kernel_io_hang, expression io_util >= 99 and io_qusize >= 20 and io_read_write_time == 0 2023-11-27 13:36:00.903 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_util', labels={}}, Metric{name='ob_host_io_read_write_time', labels={}}] 2023-11-27 13:36:00.903 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_kernel_io_hang 2023-11-27 13:36:00.903 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-os_kernel_io_hang 2023-11-27 13:36:00.903 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for monitor_exporter_unavaliable, expression monitor_exporter_avaliable == 0 2023-11-27 13:36:00.903 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitor_exporter_avaliable', labels={}}] 2023-11-27 13:36:00.903 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monitor_exporter_unavaliable 2023-11-27 13:36:00.904 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-monitor_exporter_unavaliable 2023-11-27 13:36:00.904 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_tenant_exists_expired_xa_trans, expression xa_trans_max_duration_seconds > 1200 2023-11-27 13:36:00.904 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_xa_trans_max_duration_seconds', labels={}}] 2023-11-27 13:36:00.904 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_exists_expired_xa_trans 2023-11-27 13:36:00.904 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_exists_expired_xa_trans 2023-11-27 13:36:00.904 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_compaction, expression no_compaction_seconds > 108000 2023-11-27 13:36:00.904 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_compaction_seconds', labels={}}] 2023-11-27 13:36:00.904 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_compaction 2023-11-27 13:36:00.904 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_compaction 2023-11-27 13:36:00.904 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_merge, expression no_merge_time > 108000 2023-11-27 13:36:00.922 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_merge_seconds', labels={}}] 2023-11-27 13:36:00.922 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_merge 2023-11-27 13:36:00.922 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_merge 2023-11-27 13:36:00.923 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for host_agent_res_memory_over_threshold, expression host_agent_resident_memory_gb > 1.5 2023-11-27 13:36:00.923 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_res_memory', labels={}}] 2023-11-27 13:36:00.923 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_res_memory_over_threshold 2023-11-27 13:36:00.923 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-host_agent_res_memory_over_threshold 2023-11-27 13:36:00.923 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for host_unavailable, expression host_available == 0 2023-11-27 13:36:00.923 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_available', labels={}}] 2023-11-27 13:36:00.923 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_unavailable 2023-11-27 13:36:00.923 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-host_unavailable 2023-11-27 13:36:00.923 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_overload, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps > 0 2023-11-27 13:36:00.924 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}] 2023-11-27 13:36:00.924 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_overload 2023-11-27 13:36:00.924 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_overload 2023-11-27 13:36:00.924 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_tenant_task_timeout, expression ob_tenant_task_max_duration_seconds > 10800 2023-11-27 13:36:00.924 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_task_max_duration', labels={}}] 2023-11-27 13:36:00.924 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_task_timeout 2023-11-27 13:36:00.924 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_task_timeout 2023-11-27 13:36:00.924 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_host_connection_percent_over_threshold, expression ob_connection_percent > 80 2023-11-27 13:36:00.924 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_connection_percent', labels={}}] 2023-11-27 13:36:00.924 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_connection_percent_over_threshold 2023-11-27 13:36:00.924 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_host_connection_percent_over_threshold 2023-11-27 13:36:00.925 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_host_disk_readonly, expression disk_readonly_flag == 1 2023-11-27 13:36:00.925 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_disk_readonly_flag', labels={}}] 2023-11-27 13:36:00.925 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_disk_readonly 2023-11-27 13:36:00.925 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_host_disk_readonly 2023-11-27 13:36:00.925 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 97 2023-11-27 13:36:00.925 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 95 2023-11-27 13:36:00.925 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_mem_percent', labels={}}] 2023-11-27 13:36:00.925 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_mem_percent_over_threshold 2023-11-27 13:36:00.925 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_host_mem_percent_over_threshold 2023-11-27 13:36:00.925 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_server_stopped, expression ob_server_stopped_duration_seconds > 0 2023-11-27 13:36:00.925 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_stopped_duration_seconds', labels={}}] 2023-11-27 13:36:00.926 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_stopped 2023-11-27 13:36:00.926 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_server_stopped 2023-11-27 13:36:00.926 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 100 2023-11-27 13:36:00.926 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 50 2023-11-27 13:36:00.926 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_offset_milliseconds', labels={}}] 2023-11-27 13:36:00.926 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_offset_too_large 2023-11-27 13:36:00.926 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_offset_too_large 2023-11-27 13:36:00.926 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for node_file_inode_usage, expression file_inode_usage > 80 2023-11-27 13:36:00.926 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_file_inode_usage', labels={}}] 2023-11-27 13:36:00.926 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_inode_usage 2023-11-27 13:36:00.926 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-node_file_inode_usage 2023-11-27 13:36:00.926 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_tenant_long_trans_exist, expression long_trans_max_duration_seconds > 1200 2023-11-27 13:36:00.927 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_long_trans_max_duration_seconds', labels={}}] 2023-11-27 13:36:00.927 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_long_trans_exist 2023-11-27 13:36:00.927 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_long_trans_exist 2023-11-27 13:36:00.927 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_tenant_expired_trans_exist, expression pending_trans_max_duration_seconds > 1200 2023-11-27 13:36:00.927 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_pending_trans_max_duration_seconds', labels={}}] 2023-11-27 13:36:00.927 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_expired_trans_exist 2023-11-27 13:36:00.927 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_expired_trans_exist 2023-11-27 13:36:00.927 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_tenant_large_trans_exist, expression trans_max_log_size_mb > 102400 2023-11-27 13:36:00.927 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_trans_max_log_size_mb', labels={}}] 2023-11-27 13:36:00.927 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_large_trans_exist 2023-11-27 13:36:00.927 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_large_trans_exist 2023-11-27 13:36:00.928 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for host_agent_goroutine_count_over_threshold, expression host_agent_goroutine_count > 3000 2023-11-27 13:36:00.928 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_goroutine_count', labels={}}] 2023-11-27 13:36:00.928 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_goroutine_count_over_threshold 2023-11-27 13:36:00.928 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-host_agent_goroutine_count_over_threshold 2023-11-27 13:36:00.928 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_cannot_connected, expression ob_connectable == 0 2023-11-27 13:36:00.928 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_connectable', labels={}}] 2023-11-27 13:36:00.928 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cannot_connected 2023-11-27 13:36:00.928 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_cannot_connected 2023-11-27 13:36:00.928 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_inactive_server, expression inactive_server_count > 0 2023-11-27 13:36:00.928 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_inactive_server_count', labels={}}] 2023-11-27 13:36:00.929 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_inactive_server 2023-11-27 13:36:00.929 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_inactive_server 2023-11-27 13:36:00.929 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_host_partition_count_over_threshold, expression partition_count > 30000 2023-11-27 13:36:00.929 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_partition_count', labels={}}] 2023-11-27 13:36:00.929 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_partition_count_over_threshold 2023-11-27 13:36:00.929 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_host_partition_count_over_threshold 2023-11-27 13:36:00.929 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ocp_remote_server_time_diff_too_large, expression clock_diff_milliseconds > 50 2023-11-27 13:36:00.930 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_clock_diff_milliseconds', labels={}}] 2023-11-27 13:36:00.930 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_remote_server_time_diff_too_large 2023-11-27 13:36:00.930 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ocp_remote_server_time_diff_too_large 2023-11-27 13:36:00.930 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_full, expression cpu_used_percent >= 98 2023-11-27 13:36:00.930 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_used_percent', labels={}}] 2023-11-27 13:36:00.930 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_full 2023-11-27 13:36:00.930 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_full 2023-11-27 13:36:00.930 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for host_ntp_service_not_exist, expression ntp_service_exist == 0 2023-11-27 13:36:00.930 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_service_exist', labels={}}] 2023-11-27 13:36:00.930 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_service_not_exist 2023-11-27 13:36:00.930 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_service_not_exist 2023-11-27 13:36:00.930 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_index_fail_table, expression index_fail_count > 0 2023-11-27 13:36:00.931 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_index_fail_count', labels={}}] 2023-11-27 13:36:00.931 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_index_fail_table 2023-11-27 13:36:00.931 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_index_fail_table 2023-11-27 13:36:00.931 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.a.u.f.VersionalProductFactory : refresh done, configCount=80, updatedCount=80, failedToUpdateCount=0 2023-11-27 13:36:00.938 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=3) - no updates (start=2023-11-26T13:43:30, end=2023-11-26T13:46:00) 2023-11-27 13:36:00.963 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=3) - no updates (start=2023-11-26T13:43:30, end=2023-11-26T13:46:00) 2023-11-27 13:36:00.966 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=4) - no updates (start=2023-11-26T13:46:00, end=2023-11-26T13:48:30) 2023-11-27 13:36:00.976 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=4) - no updates (start=2023-11-26T13:46:00, end=2023-11-26T13:48:30) 2023-11-27 13:36:01.008 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=4) - no updates (start=2023-11-26T13:46:00, end=2023-11-26T13:48:30) 2023-11-27 13:36:01.008 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=5) - no updates (start=2023-11-26T13:48:30, end=2023-11-26T13:51:00) 2023-11-27 13:36:01.031 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=5) - no updates (start=2023-11-26T13:48:30, end=2023-11-26T13:51:00) 2023-11-27 13:36:01.044 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=6) - no updates (start=2023-11-26T13:51:00, end=2023-11-26T13:53:30) 2023-11-27 13:36:01.065 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=14, name=zone6, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:36:01.065 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=7) - no updates (start=2023-11-26T13:53:30, end=2023-11-26T13:56:00) 2023-11-27 13:36:01.065 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=14, ip=133.197.204.13, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=7, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T20:23:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.204.13, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=7, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:23:25.656813Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:36:01.065 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=5) - no updates (start=2023-11-26T13:48:30, end=2023-11-26T13:51:00) 2023-11-27 13:36:01.075 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:01.075 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:01.076 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=6) - no updates (start=2023-11-26T13:51:00, end=2023-11-26T13:53:30) 2023-11-27 13:36:01.085 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2023-11-27 13:36:01.109 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=6) - no updates (start=2023-11-26T13:51:00, end=2023-11-26T13:53:30) 2023-11-27 13:36:01.137 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20231117 2023-11-27 13:36:01.137 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=8) - no updates (start=2023-11-26T13:56:00, end=2023-11-26T13:58:30) 2023-11-27 13:36:01.144 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=7) - no updates (start=2023-11-26T13:53:30, end=2023-11-26T13:56:00) 2023-11-27 13:36:01.145 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:01.145 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:01.169 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=7) - no updates (start=2023-11-26T13:53:30, end=2023-11-26T13:56:00) 2023-11-27 13:36:01.169 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` drop partition (P20231117) 2023-11-27 13:36:01.187 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=4) - no updates (start=2023-11-26T13:46:00, end=2023-11-26T13:48:30) 2023-11-27 13:36:01.217 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=9) - no updates (start=2023-11-26T13:58:30, end=2023-11-26T14:01:00) 2023-11-27 13:36:01.220 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=8) - no updates (start=2023-11-26T13:56:00, end=2023-11-26T13:58:30) 2023-11-27 13:36:01.486 INFO 10 --- [pool-ocp-schedules-3,,] o.s.cloud.commons.util.InetUtils : Cannot determine local hostname 2023-11-27 13:36:01.502 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:01.502 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:01.508 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2023-11-27 13:36:01.516 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=9) - no updates (start=2023-11-26T13:58:30, end=2023-11-26T14:01:00) 2023-11-27 13:36:01.518 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=8) - no updates (start=2023-11-26T13:56:00, end=2023-11-26T13:58:30) 2023-11-27 13:36:01.532 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=9) - no updates (start=2023-11-26T13:58:30, end=2023-11-26T14:01:00) 2023-11-27 13:36:01.538 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=10) - no updates (start=2023-11-26T14:01:00, end=2023-11-26T14:03:30) 2023-11-27 13:36:01.538 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=10) - no updates (start=2023-11-26T14:01:00, end=2023-11-26T14:03:30) 2023-11-27 13:36:01.545 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=11) - no updates (start=2023-11-26T14:03:30, end=2023-11-26T14:06:00) 2023-11-27 13:36:01.548 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=10) - no updates (start=2023-11-26T14:01:00, end=2023-11-26T14:03:30) 2023-11-27 13:36:01.550 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=5) - no updates (start=2023-11-26T13:48:30, end=2023-11-26T13:51:00) 2023-11-27 13:36:01.569 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20231118 2023-11-27 13:36:01.570 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:01.570 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:01.581 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=11) - no updates (start=2023-11-26T14:03:30, end=2023-11-26T14:06:00) 2023-11-27 13:36:01.582 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=11) - no updates (start=2023-11-26T14:03:30, end=2023-11-26T14:06:00) 2023-11-27 13:36:01.582 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` drop partition (P20231118) 2023-11-27 13:36:01.600 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=6) - no updates (start=2023-11-26T13:51:00, end=2023-11-26T13:53:30) 2023-11-27 13:36:01.606 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=7) - no updates (start=2023-11-26T13:53:30, end=2023-11-26T13:56:00) 2023-11-27 13:36:01.627 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=12) - no updates (start=2023-11-26T14:06:00, end=2023-11-26T14:08:30) 2023-11-27 13:36:01.627 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=12) - no updates (start=2023-11-26T14:06:00, end=2023-11-26T14:08:30) 2023-11-27 13:36:01.639 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=8) - no updates (start=2023-11-26T13:56:00, end=2023-11-26T13:58:30) 2023-11-27 13:36:01.642 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=14, ip=133.197.204.13, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=7, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:23:25.656813Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:36:01.642 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=13, name=zone5, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone5, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-27 13:36:01.642 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=13) - no updates (start=2023-11-26T14:08:30, end=2023-11-26T14:11:00) 2023-11-27 13:36:01.653 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=14) - no updates (start=2023-11-26T14:11:00, end=2023-11-26T14:13:30) 2023-11-27 13:36:01.656 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=12) - no updates (start=2023-11-26T14:06:00, end=2023-11-26T14:08:30) 2023-11-27 13:36:01.667 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=13) - no updates (start=2023-11-26T14:08:30, end=2023-11-26T14:11:00) 2023-11-27 13:36:01.668 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=9) - no updates (start=2023-11-26T13:58:30, end=2023-11-26T14:01:00) 2023-11-27 13:36:01.670 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=15) - no updates (start=2023-11-26T14:13:30, end=2023-11-26T14:16:00) 2023-11-27 13:36:01.674 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=13) - no updates (start=2023-11-26T14:08:30, end=2023-11-26T14:11:00) 2023-11-27 13:36:01.683 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=14) - no updates (start=2023-11-26T14:11:00, end=2023-11-26T14:13:30) 2023-11-27 13:36:01.690 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=14) - no updates (start=2023-11-26T14:11:00, end=2023-11-26T14:13:30) 2023-11-27 13:36:01.690 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=10) - no updates (start=2023-11-26T14:01:00, end=2023-11-26T14:03:30) 2023-11-27 13:36:01.703 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=15) - no updates (start=2023-11-26T14:13:30, end=2023-11-26T14:16:00) 2023-11-27 13:36:01.703 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=16) - no updates (start=2023-11-26T14:16:00, end=2023-11-26T14:18:30) 2023-11-27 13:36:01.717 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=11) - no updates (start=2023-11-26T14:03:30, end=2023-11-26T14:06:00) 2023-11-27 13:36:01.718 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=16) - no updates (start=2023-11-26T14:16:00, end=2023-11-26T14:18:30) 2023-11-27 13:36:01.720 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=15) - no updates (start=2023-11-26T14:13:30, end=2023-11-26T14:16:00) 2023-11-27 13:36:01.726 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=12) - no updates (start=2023-11-26T14:06:00, end=2023-11-26T14:08:30) 2023-11-27 13:36:01.730 INFO 10 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2023-11-27 13:36:01.731 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=16) - no updates (start=2023-11-26T14:16:00, end=2023-11-26T14:18:30) 2023-11-27 13:36:01.742 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=17) - no updates (start=2023-11-26T14:18:30, end=2023-11-26T14:21:00) 2023-11-27 13:36:01.745 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=17) - no updates (start=2023-11-26T14:18:30, end=2023-11-26T14:21:00) 2023-11-27 13:36:01.793 INFO 10 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2023-11-27 13:36:01.793 INFO 10 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2023-11-27 13:36:01.793 INFO 10 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : set system property ocp.version to 4.0.3 2023-11-27 13:36:01.794 INFO 10 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : set systemProperties as the first PropertySource 2023-11-27 13:36:01.794 INFO 10 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.0.3 2023-11-27 13:36:01.807 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=17) - no updates (start=2023-11-26T14:18:30, end=2023-11-26T14:21:00) 2023-11-27 13:36:01.811 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=18) - no updates (start=2023-11-26T14:21:00, end=2023-11-26T14:23:30) 2023-11-27 13:36:01.812 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=18) - no updates (start=2023-11-26T14:21:00, end=2023-11-26T14:23:30) 2023-11-27 13:36:01.824 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=18) - no updates (start=2023-11-26T14:21:00, end=2023-11-26T14:23:30) 2023-11-27 13:36:01.827 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=13) - no updates (start=2023-11-26T14:08:30, end=2023-11-26T14:11:00) 2023-11-27 13:36:01.866 WARN 10 --- [pool-ocp-schedules-3,,] ironmentPostProcessorApplicationListener : Your Okta Issuer URL is missing. You can copy your domain from the Okta Developer Console. Follow these instructions to find it: https://bit.ly/finding-okta-domain To fix this add the `okta.oauth2.issuer` property to your application environments. 2023-11-27 13:36:01.866 INFO 10 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2023-11-27 13:36:01.909 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=19) - no updates (start=2023-11-26T14:23:30, end=2023-11-26T14:26:00) 2023-11-27 13:36:01.939 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=19) - no updates (start=2023-11-26T14:23:30, end=2023-11-26T14:26:00) 2023-11-27 13:36:01.941 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=20) - no updates (start=2023-11-26T14:26:00, end=2023-11-26T14:28:30) 2023-11-27 13:36:01.957 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=13, name=zone5, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:36:01.957 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=13, ip=133.197.204.12, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=6, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T20:18:57+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.204.12, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=6, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:18:56.949667Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:36:01.962 INFO 10 --- [pool-overview-container-8,,f7d8492d18d7] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=16781 2023-11-27 13:36:01.979 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=14) - no updates (start=2023-11-26T14:11:00, end=2023-11-26T14:13:30) 2023-11-27 13:36:01.984 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=19) - no updates (start=2023-11-26T14:23:30, end=2023-11-26T14:26:00) 2023-11-27 13:36:02.010 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=21) - no updates (start=2023-11-26T14:28:30, end=2023-11-26T14:31:00) 2023-11-27 13:36:02.027 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:02.027 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:02.033 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2023-11-27 13:36:02.067 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=20) - no updates (start=2023-11-26T14:26:00, end=2023-11-26T14:28:30) 2023-11-27 13:36:02.096 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=21) - no updates (start=2023-11-26T14:28:30, end=2023-11-26T14:31:00) 2023-11-27 13:36:02.111 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=22) - no updates (start=2023-11-26T14:31:00, end=2023-11-26T14:33:30) 2023-11-27 13:36:02.134 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=15) - no updates (start=2023-11-26T14:13:30, end=2023-11-26T14:16:00) 2023-11-27 13:36:02.135 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:02.135 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:02.149 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:02.157 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:02.157 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:02.158 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=20) - no updates (start=2023-11-26T14:26:00, end=2023-11-26T14:28:30) 2023-11-27 13:36:02.161 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2023-11-27 13:36:02.169 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=16) - no updates (start=2023-11-26T14:16:00, end=2023-11-26T14:18:30) 2023-11-27 13:36:02.179 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=22) - no updates (start=2023-11-26T14:31:00, end=2023-11-26T14:33:30) 2023-11-27 13:36:02.202 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=21) - no updates (start=2023-11-26T14:28:30, end=2023-11-26T14:31:00) 2023-11-27 13:36:02.211 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20231203 2023-11-27 13:36:02.215 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:02.215 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:02.224 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20231203 values less than (1701648000000000)) 2023-11-27 13:36:02.234 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=23) - no updates (start=2023-11-26T14:33:30, end=2023-11-26T14:36:00) 2023-11-27 13:36:02.242 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=17) - no updates (start=2023-11-26T14:18:30, end=2023-11-26T14:21:00) 2023-11-27 13:36:02.244 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=22) - no updates (start=2023-11-26T14:31:00, end=2023-11-26T14:33:30) 2023-11-27 13:36:02.270 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=24) - no updates (start=2023-11-26T14:36:00, end=2023-11-26T14:38:30) 2023-11-27 13:36:02.343 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=18) - no updates (start=2023-11-26T14:21:00, end=2023-11-26T14:23:30) 2023-11-27 13:36:02.344 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=25) - no updates (start=2023-11-26T14:38:30, end=2023-11-26T14:41:00) 2023-11-27 13:36:02.433 INFO 10 --- [pool-scanner--161-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=132 2023-11-27 13:36:02.442 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=13, ip=133.197.204.12, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=6, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:18:56.949667Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:36:02.442 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=12, name=zone4, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone4, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-27 13:36:02.526 INFO 10 --- [pool-ocp-schedules-7,,7214872cc100] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:36:02.594 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=23) - no updates (start=2023-11-26T14:33:30, end=2023-11-26T14:36:00) 2023-11-27 13:36:02.609 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:02.609 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:02.619 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=23) - no updates (start=2023-11-26T14:33:30, end=2023-11-26T14:36:00) 2023-11-27 13:36:02.656 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=26) - no updates (start=2023-11-26T14:41:00, end=2023-11-26T14:43:30) 2023-11-27 13:36:02.658 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=19) - no updates (start=2023-11-26T14:23:30, end=2023-11-26T14:26:00) 2023-11-27 13:36:02.683 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2023-11-27 13:36:02.684 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=24) - no updates (start=2023-11-26T14:36:00, end=2023-11-26T14:38:30) 2023-11-27 13:36:02.758 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2023-11-27 13:36:02.765 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=24) - no updates (start=2023-11-26T14:36:00, end=2023-11-26T14:38:30) 2023-11-27 13:36:02.765 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20231204 2023-11-27 13:36:02.769 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:02.769 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:02.779 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20231204 values less than (1701734400000000)) 2023-11-27 13:36:02.789 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=25) - no updates (start=2023-11-26T14:38:30, end=2023-11-26T14:41:00) 2023-11-27 13:36:02.803 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=20) - no updates (start=2023-11-26T14:26:00, end=2023-11-26T14:28:30) 2023-11-27 13:36:02.831 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=26) - no updates (start=2023-11-26T14:41:00, end=2023-11-26T14:43:30) 2023-11-27 13:36:02.839 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=25) - no updates (start=2023-11-26T14:38:30, end=2023-11-26T14:41:00) 2023-11-27 13:36:02.851 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=27) - no updates (start=2023-11-26T14:43:30, end=2023-11-26T14:46:00) 2023-11-27 13:36:02.862 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=21) - no updates (start=2023-11-26T14:28:30, end=2023-11-26T14:31:00) 2023-11-27 13:36:03.316 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=22) - no updates (start=2023-11-26T14:31:00, end=2023-11-26T14:33:30) 2023-11-27 13:36:03.318 INFO 10 --- [pool-prometheus-second-schedule-2,,01f4eabfeda4] c.o.o.m.helper.ExporterRequestHelper : Get http client too slow, exporterAddr=OcpExporterAddr(id=2000114, exporterType=OB_CLUSTER, jobName=ob, hostId=1, instance=http://133.197.206.97:62889/metrics/ob/basic, scrapeInterval=1, status=active, gmtCreate=2023-04-25 14:54:57.0, gmtModified=2023-11-13 05:13:13.0, uri=/ob/basic, path=/metrics/ob/basic, port=62889), start=1701063355162, elapsed=8156 2023-11-27 13:36:03.320 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=27) - no updates (start=2023-11-26T14:43:30, end=2023-11-26T14:46:00) 2023-11-27 13:36:03.321 INFO 10 --- [pool-prometheus-second-schedule-1,,44a067541436] c.o.o.m.helper.ExporterRequestHelper : Get http client too slow, exporterAddr=OcpExporterAddr(id=2000112, exporterType=HOST_MONITOR, jobName=ob, hostId=1, instance=http://133.197.206.97:62889/metrics/node/host, scrapeInterval=1, status=active, gmtCreate=2023-04-25 14:54:57.0, gmtModified=2023-11-13 00:06:16.0, uri=/node/host, path=/metrics/node/host, port=62889), start=1701063355160, elapsed=8161 2023-11-27 13:36:03.323 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=28) - no updates (start=2023-11-26T14:46:00, end=2023-11-26T14:48:30) 2023-11-27 13:36:03.325 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=26) - no updates (start=2023-11-26T14:41:00, end=2023-11-26T14:43:30) 2023-11-27 13:36:03.330 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=23) - no updates (start=2023-11-26T14:33:30, end=2023-11-26T14:36:00) 2023-11-27 13:36:03.351 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=28) - no updates (start=2023-11-26T14:46:00, end=2023-11-26T14:48:30) 2023-11-27 13:36:03.352 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=24) - no updates (start=2023-11-26T14:36:00, end=2023-11-26T14:38:30) 2023-11-27 13:36:03.356 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=27) - no updates (start=2023-11-26T14:43:30, end=2023-11-26T14:46:00) 2023-11-27 13:36:03.367 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=29) - no updates (start=2023-11-26T14:48:30, end=2023-11-26T14:51:00) 2023-11-27 13:36:03.368 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=12, name=zone4, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:36:03.368 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=12, ip=133.197.204.11, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T20:11:45+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.204.11, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:11:45.487530Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:36:03.381 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=28) - no updates (start=2023-11-26T14:46:00, end=2023-11-26T14:48:30) 2023-11-27 13:36:03.393 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=25) - no updates (start=2023-11-26T14:38:30, end=2023-11-26T14:41:00) 2023-11-27 13:36:03.406 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=29) - no updates (start=2023-11-26T14:48:30, end=2023-11-26T14:51:00) 2023-11-27 13:36:03.413 INFO 10 --- [pool-ocp-schedules-11,,6d1f3280bb58] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, newSeriesIdsCount=20000, maxSyncedSeriesId=514632610623913984 2023-11-27 13:36:03.415 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=30) - no updates (start=2023-11-26T14:51:00, end=2023-11-26T14:53:30) 2023-11-27 13:36:03.416 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=29) - no updates (start=2023-11-26T14:48:30, end=2023-11-26T14:51:00) 2023-11-27 13:36:03.422 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=26) - no updates (start=2023-11-26T14:41:00, end=2023-11-26T14:43:30) 2023-11-27 13:36:03.437 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:36:03.449 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=30) - no updates (start=2023-11-26T14:51:00, end=2023-11-26T14:53:30) 2023-11-27 13:36:03.449 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=31) - no updates (start=2023-11-26T14:53:30, end=2023-11-26T14:56:00) 2023-11-27 13:36:03.450 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=30) - no updates (start=2023-11-26T14:51:00, end=2023-11-26T14:53:30) 2023-11-27 13:36:03.476 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:36:03.481 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:03.481 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:03.484 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2023-11-27 13:36:03.486 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:36:03.483800+08:00, lastProcessTime=2023-11-27T13:35:55.407106+08:00 2023-11-27 13:36:03.514 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:36:03.543 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2023-11-27 13:36:03.547 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20231205 2023-11-27 13:36:03.547 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:03.547 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:03.548 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20231205 values less than (1701820800000000)) 2023-11-27 13:36:03.552 INFO 10 --- [pool-prometheus-second-schedule-1,,44a067541436] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=3, collectTaskCount=3 2023-11-27 13:36:03.552 INFO 10 --- [pool-prometheus-second-schedule-1,,44a067541436] c.o.ocp.monitor.OcpMonitorManager : Collect schedule too slow, from=1701063355160, count=3, elapsed=8392 2023-11-27 13:36:03.582 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=31) - no updates (start=2023-11-26T14:53:30, end=2023-11-26T14:56:00) 2023-11-27 13:36:03.582 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=31) - no updates (start=2023-11-26T14:53:30, end=2023-11-26T14:56:00) 2023-11-27 13:36:03.607 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=32) - no updates (start=2023-11-26T14:56:00, end=2023-11-26T14:58:30) 2023-11-27 13:36:03.608 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=32) - no updates (start=2023-11-26T14:56:00, end=2023-11-26T14:58:30) 2023-11-27 13:36:03.609 INFO 10 --- [pool-prometheus-second-schedule-2,,01f4eabfeda4] c.o.o.m.helper.ExporterRequestHelper : Get http client too slow, exporterAddr=OcpExporterAddr(id=2000161, exporterType=HOST_MONITOR, jobName=ob, hostId=9, instance=http://133.197.206.88:62889/metrics/node/host, scrapeInterval=1, status=active, gmtCreate=2023-04-27 14:57:20.0, gmtModified=2023-10-11 07:13:39.0, uri=/node/host, path=/metrics/node/host, port=62889), start=1701063363362, elapsed=247 2023-11-27 13:36:03.609 INFO 10 --- [pool-prometheus-second-schedule-2,,01f4eabfeda4] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=2, collectTaskCount=2 2023-11-27 13:36:03.609 INFO 10 --- [pool-prometheus-second-schedule-2,,01f4eabfeda4] c.o.ocp.monitor.OcpMonitorManager : Collect schedule too slow, from=1701063355161, count=2, elapsed=8448 2023-11-27 13:36:03.612 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=27) - no updates (start=2023-11-26T14:43:30, end=2023-11-26T14:46:00) 2023-11-27 13:36:03.633 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=32) - no updates (start=2023-11-26T14:56:00, end=2023-11-26T14:58:30) 2023-11-27 13:36:03.668 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=33) - no updates (start=2023-11-26T14:58:30, end=2023-11-26T15:01:00) 2023-11-27 13:36:03.671 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=33) - no updates (start=2023-11-26T14:58:30, end=2023-11-26T15:01:00) 2023-11-27 13:36:03.671 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:36:03.673 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=33) - no updates (start=2023-11-26T14:58:30, end=2023-11-26T15:01:00) 2023-11-27 13:36:03.709 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=28) - no updates (start=2023-11-26T14:46:00, end=2023-11-26T14:48:30) 2023-11-27 13:36:03.721 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=34) - no updates (start=2023-11-26T15:01:00, end=2023-11-26T15:03:30) 2023-11-27 13:36:03.723 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:36:03.747 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=34) - no updates (start=2023-11-26T15:01:00, end=2023-11-26T15:03:30) 2023-11-27 13:36:03.749 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=29) - no updates (start=2023-11-26T14:48:30, end=2023-11-26T14:51:00) 2023-11-27 13:36:03.782 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=34) - no updates (start=2023-11-26T15:01:00, end=2023-11-26T15:03:30) 2023-11-27 13:36:03.868 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=35) - no updates (start=2023-11-26T15:03:30, end=2023-11-26T15:06:00) 2023-11-27 13:36:03.906 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=12, ip=133.197.204.11, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:11:45.487530Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:36:03.906 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=11, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-27 13:36:03.908 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=36) - no updates (start=2023-11-26T15:06:00, end=2023-11-26T15:08:30) 2023-11-27 13:36:03.967 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=35) - no updates (start=2023-11-26T15:03:30, end=2023-11-26T15:06:00) 2023-11-27 13:36:04.014 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=35) - no updates (start=2023-11-26T15:03:30, end=2023-11-26T15:06:00) 2023-11-27 13:36:04.014 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=30) - no updates (start=2023-11-26T14:51:00, end=2023-11-26T14:53:30) 2023-11-27 13:36:04.027 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:04.027 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:04.038 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=36) - no updates (start=2023-11-26T15:06:00, end=2023-11-26T15:08:30) 2023-11-27 13:36:04.045 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2023-11-27 13:36:04.011 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=36) - no updates (start=2023-11-26T15:06:00, end=2023-11-26T15:08:30) 2023-11-27 13:36:04.005 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=37) - no updates (start=2023-11-26T15:08:30, end=2023-11-26T15:11:00) 2023-11-27 13:36:04.096 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=11, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:36:04.097 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=11, ip=133.197.204.10, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=true, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T20:08:12+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.204.10, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=true, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:08:12.226557Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:36:04.098 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=37) - no updates (start=2023-11-26T15:08:30, end=2023-11-26T15:11:00) 2023-11-27 13:36:04.100 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=31) - no updates (start=2023-11-26T14:53:30, end=2023-11-26T14:56:00) 2023-11-27 13:36:04.104 INFO 10 --- [pool-ocp-schedules-11,,6d1f3280bb58] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, newSeriesIdsCount=7143, maxSyncedSeriesId=584494706496503808 2023-11-27 13:36:04.110 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=38) - no updates (start=2023-11-26T15:11:00, end=2023-11-26T15:13:30) 2023-11-27 13:36:04.112 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=37) - no updates (start=2023-11-26T15:08:30, end=2023-11-26T15:11:00) 2023-11-27 13:36:04.133 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=38) - no updates (start=2023-11-26T15:11:00, end=2023-11-26T15:13:30) 2023-11-27 13:36:04.133 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=39) - no updates (start=2023-11-26T15:13:30, end=2023-11-26T15:16:00) 2023-11-27 13:36:04.139 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=32) - no updates (start=2023-11-26T14:56:00, end=2023-11-26T14:58:30) 2023-11-27 13:36:04.143 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=38) - no updates (start=2023-11-26T15:11:00, end=2023-11-26T15:13:30) 2023-11-27 13:36:04.153 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=40) - no updates (start=2023-11-26T15:16:00, end=2023-11-26T15:18:30) 2023-11-27 13:36:04.154 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=39) - no updates (start=2023-11-26T15:13:30, end=2023-11-26T15:16:00) 2023-11-27 13:36:04.171 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=39) - no updates (start=2023-11-26T15:13:30, end=2023-11-26T15:16:00) 2023-11-27 13:36:04.189 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=33) - no updates (start=2023-11-26T14:58:30, end=2023-11-26T15:01:00) 2023-11-27 13:36:04.197 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdCleanService : Delete seriesKeyIds, size=77 2023-11-27 13:36:04.198 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=41) - no updates (start=2023-11-26T15:18:30, end=2023-11-26T15:21:00) 2023-11-27 13:36:04.190 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2023-11-27 13:36:04.235 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:36:04.235 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:36:04.245 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20231116 2023-11-27 13:36:04.248 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=40) - no updates (start=2023-11-26T15:16:00, end=2023-11-26T15:18:30) 2023-11-27 13:36:04.253 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:04.253 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:04.256 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=34) - no updates (start=2023-11-26T15:01:00, end=2023-11-26T15:03:30) 2023-11-27 13:36:04.270 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` drop partition (P20231116) 2023-11-27 13:36:04.296 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=42) - no updates (start=2023-11-26T15:21:00, end=2023-11-26T15:23:30) 2023-11-27 13:36:04.311 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=41) - no updates (start=2023-11-26T15:18:30, end=2023-11-26T15:21:00) 2023-11-27 13:36:04.313 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=35) - no updates (start=2023-11-26T15:03:30, end=2023-11-26T15:06:00) 2023-11-27 13:36:04.316 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=40) - no updates (start=2023-11-26T15:16:00, end=2023-11-26T15:18:30) 2023-11-27 13:36:04.342 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063355, message=java.util.concurrent.TimeoutException: Request timeout to 133.197.206.97:62889 after 2000 ms, rootCause=TimeoutException: Request timeout to 133.197.206.97:62889 after 2000 ms 2023-11-27 13:36:04.342 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063355, message=java.util.concurrent.TimeoutException: Request timeout to 133.197.206.97:62889 after 2000 ms, rootCause=TimeoutException: Request timeout to 133.197.206.97:62889 after 2000 ms 2023-11-27 13:36:04.348 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=11, ip=133.197.204.10, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=true, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:08:12.226557Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:36:04.348 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-27 13:36:04.359 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=41) - no updates (start=2023-11-26T15:18:30, end=2023-11-26T15:21:00) 2023-11-27 13:36:04.366 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=36) - no updates (start=2023-11-26T15:06:00, end=2023-11-26T15:08:30) 2023-11-27 13:36:04.388 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_host_partition_count_over_threshold, id=584494695629062144 2023-11-27 13:36:04.388 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_host_partition_count_over_threshold, id=584494695738114048 2023-11-27 13:36:04.388 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=os_observer_not_exist, id=584494695851360256 2023-11-27 13:36:04.388 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=os_observer_not_exist, id=584494695897497600 2023-11-27 13:36:04.388 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_tenant500_mem_hold_percent_over_threshold, id=584494695952023552 2023-11-27 13:36:04.388 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_tenant500_mem_hold_percent_over_threshold, id=584494696027521024 2023-11-27 13:36:04.388 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_tenant_log_disk_usage_high, id=584494696056881152 2023-11-27 13:36:04.388 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_tenant_log_disk_usage_high, id=584494696073658368 2023-11-27 13:36:04.388 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=obproxyd_process_dead, id=584494696082046976 2023-11-27 13:36:04.388 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=obproxyd_process_dead, id=584494696090435584 2023-11-27 13:36:04.388 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=os_tsar_traffic_overload, id=584494696098824192 2023-11-27 13:36:04.388 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=os_tsar_traffic_overload, id=584494696107212800 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=host_ntp_service_not_exist, id=584494696119795712 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=host_ntp_service_not_exist, id=584494696165933056 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ocp_http_request_timeout, id=584494696182710272 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ocp_http_request_timeout, id=584494696224653312 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_tenant_expired_trans_exist, id=584494696266596352 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_tenant_expired_trans_exist, id=584494696279179264 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ocp_remote_server_time_diff_too_large, id=584494696291762176 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ocp_remote_server_time_diff_too_large, id=584494696363065344 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_tenant_no_frozen, id=584494696379842560 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_tenant_no_frozen, id=584494696497283072 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_tenant500_mem_hold_over_threshold, id=584494696509865984 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_tenant500_mem_hold_over_threshold, id=584494696530837504 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=host_disk_readonly, id=584494696706998272 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=host_disk_readonly, id=584494696799272960 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=tenant_memstore_percent_over_threshold, id=584494696912519168 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=tenant_memstore_percent_over_threshold, id=584494696950267904 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_cannot_connected, id=584494697046736896 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_cannot_connected, id=584494697101262848 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_host_log_disk_percent_over_threshold, id=584494697118040064 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_host_log_disk_percent_over_threshold, id=584494697139011584 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=os_tsar_sda_ioawait, id=584494697180954624 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=os_tsar_sda_ioawait, id=584494697252257792 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_cluster_sync_failed, id=584494697340338176 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_cluster_sync_failed, id=584494697369698304 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=obagent_dead, id=584494697386475520 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=obagent_dead, id=584494697415835648 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=os_observer_core_dump, id=584494697533276160 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=os_observer_core_dump, id=584494697600385024 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_host_net_send_percent_over_threshold, id=584494697680076800 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_host_net_send_percent_over_threshold, id=584494697684271104 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_zone_sstable_percent_over_threshold, id=584494697692659712 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_zone_sstable_percent_over_threshold, id=584494697717825536 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_cluster_merge_timeout, id=584494697726214144 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_cluster_merge_timeout, id=584494697768157184 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=obproxy_process_dead, id=584494697776545792 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=obproxy_process_dead, id=584494697789128704 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=os_tsar_traffic_drop, id=584494697801711616 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=os_tsar_traffic_drop, id=584494697852043264 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ocp_alarm_detect_timeout, id=584494697864626176 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ocp_alarm_detect_timeout, id=584494697868820480 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_cluster_merge_error, id=584494697877209088 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_cluster_merge_error, id=584494697881403392 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_server_stopped, id=584494697893986304 2023-11-27 13:36:04.389 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_server_stopped, id=584494697902374912 2023-11-27 13:36:04.390 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=node_file_inode_usage, id=584494697910763520 2023-11-27 13:36:04.390 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=node_file_inode_usage, id=584494697914957824 2023-11-27 13:36:04.390 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_host_data_disk_percent_over_threshold, id=584494697923346432 2023-11-27 13:36:04.390 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_host_data_disk_percent_over_threshold, id=584494697940123648 2023-11-27 13:36:04.390 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=os_tsar_cpu_util_full, id=584494697944317952 2023-11-27 13:36:04.390 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=os_tsar_cpu_util_full, id=584494697952706560 2023-11-27 13:36:04.390 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_tenant_compaction_error, id=584494697961095168 2023-11-27 13:36:04.390 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_tenant_compaction_error, id=584494697965289472 2023-11-27 13:36:04.390 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_cluster_no_frozen, id=584494697977872384 2023-11-27 13:36:04.390 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_cluster_no_frozen, id=584494697986260992 2023-11-27 13:36:04.390 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=monitor_exporter_unavaliable, id=584494697994649600 2023-11-27 13:36:04.390 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=monitor_exporter_unavaliable, id=584494698003038208 2023-11-27 13:36:04.390 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_tenant_exists_expired_xa_trans, id=584494698015621120 2023-11-27 13:36:04.390 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_tenant_exists_expired_xa_trans, id=584494698032398336 2023-11-27 13:36:04.390 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_cluster_exists_inactive_server, id=584494698040786944 2023-11-27 13:36:04.390 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_cluster_exists_inactive_server, id=584494698091118592 2023-11-27 13:36:04.390 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_host_cpu_percent_over_threshold, id=584494698095312896 2023-11-27 13:36:04.390 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_host_cpu_percent_over_threshold, id=584494698103701504 2023-11-27 13:36:04.390 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ocp_http_request_too_many_errors_occur, id=584494698120478720 2023-11-27 13:36:04.390 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ocp_http_request_too_many_errors_occur, id=584494698128867328 2023-11-27 13:36:04.390 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=os_tsar_cpu_util_hwm, id=584494698133061632 2023-11-27 13:36:04.391 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=43) - no updates (start=2023-11-26T15:23:30, end=2023-11-26T15:26:00) 2023-11-27 13:36:04.412 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=42) - no updates (start=2023-11-26T15:21:00, end=2023-11-26T15:23:30) 2023-11-27 13:36:04.413 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdCleanService : Delete seriesKeyIds, size=143 2023-11-27 13:36:04.415 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=37) - no updates (start=2023-11-26T15:08:30, end=2023-11-26T15:11:00) 2023-11-27 13:36:04.427 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:36:03.483800+08:00 2023-11-27 13:36:04.430 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=42) - no updates (start=2023-11-26T15:21:00, end=2023-11-26T15:23:30) 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=os_tsar_cpu_util_hwm, id=584494698141450240 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_host_disk_readonly, id=584494698149838848 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_host_disk_readonly, id=584494698166616064 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ocp_meta_db_disconnected, id=584494698175004672 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ocp_meta_db_disconnected, id=584494698183393280 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_host_load1_per_cpu_over_threshold, id=584494698191781888 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_host_load1_per_cpu_over_threshold, id=584494698200170496 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=os_cpu_irq_error, id=584494698221142016 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=os_cpu_irq_error, id=584494698225336320 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_host_net_recv_percent_over_threshold, id=584494698233724928 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_host_net_recv_percent_over_threshold, id=584494698242113536 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=host_agent_open_fd_count_over_threshold, id=584494698271473664 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=host_agent_open_fd_count_over_threshold, id=584494698284056576 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_host_mem_percent_over_threshold, id=584494698300833792 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_host_mem_percent_over_threshold, id=584494698313416704 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_cluster_no_merge, id=584494698325999616 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_cluster_no_merge, id=584494698338582528 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=host_agent_res_memory_over_threshold, id=584494698346971136 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=host_agent_res_memory_over_threshold, id=584494698401497088 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=backup_storage_capacity_retry_times_exceeded, id=584494698439245824 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=backup_storage_capacity_retry_times_exceeded, id=584494698560880640 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=os_kernel_io_hang, id=584494698661543936 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=os_kernel_io_hang, id=584494698716069888 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=os_observer_fd_usage, id=584494698841899008 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=os_observer_fd_usage, id=584494698917396480 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=node_file_root_usage, id=584494698980311040 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=node_file_root_usage, id=584494699110334464 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_tenant_task_timeout, id=584494699206803456 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_tenant_task_timeout, id=584494699328438272 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_host_tcp_retrans_percent_over_threshold, id=584494699462656000 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_host_tcp_retrans_percent_over_threshold, id=584494699601068032 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_cluster_sync_delay_time_too_long, id=584494699701731328 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_cluster_sync_delay_time_too_long, id=584494699814977536 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=os_nic_1000m_full, id=584494699890475008 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=os_nic_1000m_full, id=584494699965972480 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_host_install_disk_percent_over_threshold, id=584494699995332608 2023-11-27 13:36:04.440 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_host_install_disk_percent_over_threshold, id=584494700007915520 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ic_server_connect_failed, id=584494700016304128 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ic_server_connect_failed, id=584494700133744640 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=os_tsar_nvme_ioawait, id=584494700226019328 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=os_tsar_nvme_ioawait, id=584494700284739584 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_cluster_status_check_failed, id=584494700309905408 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_cluster_status_check_failed, id=584494700381208576 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_host_connection_percent_over_threshold, id=584494700423151616 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_host_connection_percent_over_threshold, id=584494700599312384 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=host_agent_goroutine_count_over_threshold, id=584494700762890240 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=host_agent_goroutine_count_over_threshold, id=584494700804833280 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_server_sstable_percent_over_threshold, id=584494700905496576 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_server_sstable_percent_over_threshold, id=584494700930662400 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_tenant_long_trans_exist, id=584494700972605440 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_tenant_long_trans_exist, id=584494701043908608 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_host_task_timeout, id=584494701069074432 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_host_task_timeout, id=584494701358481408 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=obproxy_cannot_connected, id=584494701417201664 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=obproxy_cannot_connected, id=584494701496893440 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=os_tsar_cpu_sys_abnormal, id=584494701538836480 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=os_tsar_cpu_sys_abnormal, id=584494701631111168 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=tenant_active_memstore_percent_over_threshold, id=584494701681442816 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=tenant_active_memstore_percent_over_threshold, id=584494701698220032 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_cluster_exists_index_fail_table, id=584494701773717504 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_cluster_exists_index_fail_table, id=584494701861797888 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=tenant_cpu_percent_over_threshold, id=584494701912129536 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=tenant_cpu_percent_over_threshold, id=584494701987627008 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=os_nic_1000m_hwm, id=584494702037958656 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=os_nic_1000m_hwm, id=584494702063124480 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=os_tsar_traffic_error, id=584494702096678912 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=os_tsar_traffic_error, id=584494702117650432 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_cluster_frozen_version_delta_over_threshold, id=584494702142816256 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_cluster_frozen_version_delta_over_threshold, id=584494702214119424 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=host_ntp_offset_too_large, id=584494702281228288 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=host_ntp_offset_too_large, id=584494702310588416 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_cpu_percent_over_threshold, id=584494702331559936 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_cpu_percent_over_threshold, id=584494702394474496 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=vpc_connect_failed, id=584494702432223232 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=vpc_connect_failed, id=584494702507720704 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=backup_storage_capacity_timeout_or_interrupted, id=584494702570635264 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=backup_storage_capacity_timeout_or_interrupted, id=584494702595801088 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_tenant_large_trans_exist, id=584494702650327040 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_tenant_large_trans_exist, id=584494702671298560 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=ob_tenant_no_compaction, id=584494702683881472 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=ob_tenant_no_compaction, id=584494702725824512 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_count|app=HOST|svr_ip=10.120.0.1|type=host_unavailable, id=584494702822293504 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_alarm_detect_duration_ms_sum|app=HOST|svr_ip=10.120.0.1|type=host_unavailable, id=584494702855847936 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_meta_db_health|app=HOST|db=meta|svr_ip=10.120.0.1, id=584494702876819456 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=ocp_meta_db_health|app=HOST|db=monitor|svr_ip=10.120.0.1, id=584494702927151104 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_count|app=HOST|exception=None|method=POST|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=root, id=584494703023620096 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_sum|app=HOST|exception=None|method=POST|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=root, id=584494703078146048 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_count|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/compute/regions, id=584494703136866304 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_sum|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/compute/regions, id=584494703187197952 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_count|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/alarm/templates, id=584494703241723904 2023-11-27 13:36:04.441 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_sum|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/alarm/templates, id=584494703262695424 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_count|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/time, id=584494703363358720 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_sum|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/time, id=584494703380135936 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_count|app=HOST|exception=None|method=GET|outcome=REDIRECTION|status=302|svr_ip=10.120.0.1|uri=REDIRECTION, id=584494703422078976 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_sum|app=HOST|exception=None|method=GET|outcome=REDIRECTION|status=302|svr_ip=10.120.0.1|uri=REDIRECTION, id=584494703484993536 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_count|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/maz/state, id=584494703505965056 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_sum|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/maz/state, id=584494703539519488 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_count|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/ob/tenants, id=584494703564685312 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_sum|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/ob/tenants, id=584494703602434048 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_count|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/mc/info, id=584494703644377088 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_sum|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/mc/info, id=584494703698903040 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_count|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/compute/hosts, id=584494703719874560 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_sum|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/compute/hosts, id=584494703745040384 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_count|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/compute/stats/hostByIdc, id=584494703782789120 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_sum|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/compute/stats/hostByIdc, id=584494703807954944 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_count|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/ob/clusters/stats/topAvgCompactionDuration, id=584494703845703680 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_sum|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/ob/clusters/stats/topAvgCompactionDuration, id=584494703879258112 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_count|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/compute/idcs, id=584494703900229632 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_sum|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/compute/idcs, id=584494703908618240 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_count|app=HOST|exception=None|method=GET|outcome=CLIENT_ERROR|status=401|svr_ip=10.120.0.1|uri=root, id=584494703912812544 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_sum|app=HOST|exception=None|method=GET|outcome=CLIENT_ERROR|status=401|svr_ip=10.120.0.1|uri=root, id=584494703921201152 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_count|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/ob/clusters, id=584494703992504320 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_sum|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/ob/clusters, id=584494704038641664 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_count|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/monitor/top, id=584494704080584704 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_sum|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/monitor/top, id=584494704101556224 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_count|app=HOST|exception=JpaSystemException|method=GET|outcome=SERVER_ERROR|status=500|svr_ip=10.120.0.1|uri=root, id=584494704147693568 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_sum|app=HOST|exception=JpaSystemException|method=GET|outcome=SERVER_ERROR|status=500|svr_ip=10.120.0.1|uri=root, id=584494704193830912 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_count|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/loginKey, id=584494704231579648 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_sum|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/loginKey, id=584494704265134080 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_count|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/monitor/metricGroups, id=584494704294494208 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_sum|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/monitor/metricGroups, id=584494704382574592 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_count|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/services, id=584494704466460672 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_sum|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/services, id=584494704512598016 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_count|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/config/systemInfo, id=584494704609067008 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_sum|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/config/systemInfo, id=584494704713924608 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_count|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/ob/clusters/stats, id=584494704760061952 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_sum|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/ob/clusters/stats, id=584494704869113856 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_count|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/profiles/me, id=584494704982360064 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_sum|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/profiles/me, id=584494705083023360 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_count|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/alarm/alarms, id=584494705192075264 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_sum|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/alarm/alarms, id=584494705343070208 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_count|app=HOST|exception=None|method=POST|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/services, id=584494705380818944 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_sum|app=HOST|exception=None|method=POST|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/services, id=584494705598922752 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_count|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/ob/clusters/tenants/topAvgCompactionDuration, id=584494705695391744 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_sum|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/ob/clusters/tenants/topAvgCompactionDuration, id=584494705837998080 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_count|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/info, id=584494705934467072 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_sum|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/info, id=584494706005770240 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_count|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/obproxy/clusters, id=584494706068684800 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_sum|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/api/v2/obproxy/clusters, id=584494706173542400 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_count|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/**, id=584494706253234176 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_sum|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/**, id=584494706341314560 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_count|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/login, id=584494706425200640 2023-11-27 13:36:04.442 INFO 10 --- [pool-ocp-schedules-4,,e0249181793c] c.o.o.m.service.SeriesIdKeyService : Clean series key id, key=http_server_requests_seconds_sum|app=HOST|exception=None|method=GET|outcome=SUCCESS|status=200|svr_ip=10.120.0.1|uri=/login, id=584494706496503808 2023-11-27 13:36:04.451 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=44) - no updates (start=2023-11-26T15:26:00, end=2023-11-26T15:28:30) 2023-11-27 13:36:04.455 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=38) - no updates (start=2023-11-26T15:11:00, end=2023-11-26T15:13:30) 2023-11-27 13:36:04.459 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=43) - no updates (start=2023-11-26T15:23:30, end=2023-11-26T15:26:00) 2023-11-27 13:36:04.480 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=43) - no updates (start=2023-11-26T15:23:30, end=2023-11-26T15:26:00) 2023-11-27 13:36:04.509 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=39) - no updates (start=2023-11-26T15:13:30, end=2023-11-26T15:16:00) 2023-11-27 13:36:04.511 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=45) - no updates (start=2023-11-26T15:28:30, end=2023-11-26T15:31:00) 2023-11-27 13:36:04.511 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=44) - no updates (start=2023-11-26T15:26:00, end=2023-11-26T15:28:30) 2023-11-27 13:36:04.538 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=44) - no updates (start=2023-11-26T15:26:00, end=2023-11-26T15:28:30) 2023-11-27 13:36:04.548 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=40) - no updates (start=2023-11-26T15:16:00, end=2023-11-26T15:18:30) 2023-11-27 13:36:04.551 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=45) - no updates (start=2023-11-26T15:28:30, end=2023-11-26T15:31:00) 2023-11-27 13:36:04.594 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=46) - no updates (start=2023-11-26T15:31:00, end=2023-11-26T15:33:30) 2023-11-27 13:36:04.604 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=45) - no updates (start=2023-11-26T15:28:30, end=2023-11-26T15:31:00) 2023-11-27 13:36:04.621 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:04.621 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:04.624 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2023-11-27 13:36:04.647 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=47) - no updates (start=2023-11-26T15:33:30, end=2023-11-26T15:36:00) 2023-11-27 13:36:04.649 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=46) - no updates (start=2023-11-26T15:31:00, end=2023-11-26T15:33:30) 2023-11-27 13:36:04.650 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=41) - no updates (start=2023-11-26T15:18:30, end=2023-11-26T15:21:00) 2023-11-27 13:36:04.655 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=46) - no updates (start=2023-11-26T15:31:00, end=2023-11-26T15:33:30) 2023-11-27 13:36:04.656 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:36:04.656 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=133.197.204.3, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=2, status=UNAVAILABLE, innerStatus=INACTIVE, startTime=2023-09-26T20:33:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.204.3, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=2, status=null, innerStatus=INACTIVE, startTime=1970-01-01T00:00Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:36:04.661 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=48) - no updates (start=2023-11-26T15:36:00, end=2023-11-26T15:38:30) 2023-11-27 13:36:04.679 INFO 10 --- [http-nio-0.0.0.0-8080-exec-6,04e6e081c6e64ed5,e9faad245b90] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.11, traceId=04e6e081c6e64ed5, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:36:04.691 INFO 10 --- [SimpleAsyncTaskExecutor-11,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:04.695 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=42) - no updates (start=2023-11-26T15:21:00, end=2023-11-26T15:23:30) 2023-11-27 13:36:04.696 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=47) - no updates (start=2023-11-26T15:33:30, end=2023-11-26T15:36:00) 2023-11-27 13:36:04.696 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=49) - no updates (start=2023-11-26T15:38:30, end=2023-11-26T15:41:00) 2023-11-27 13:36:04.699 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=47) - no updates (start=2023-11-26T15:33:30, end=2023-11-26T15:36:00) 2023-11-27 13:36:04.714 INFO 10 --- [SimpleAsyncTaskExecutor-11,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:04.722 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:36:04.715 INFO 10 --- [SimpleAsyncTaskExecutor-11,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:04.726 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=48) - no updates (start=2023-11-26T15:36:00, end=2023-11-26T15:38:30) 2023-11-27 13:36:04.727 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup delay (cluster=darenOB:10, server=4, level=0->1) - 1312 min 2023-11-27 13:36:04.727 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=darenOB:10, server=4, level=1->2, c=0) - fall behind, will continue from 2023-11-26T13:36:00 (source=2023-11-26T15:41:00, target=2023-11-24T10:17:53, granule=600000ms, delay=192187000ms) 2023-11-27 13:36:04.737 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=43) - no updates (start=2023-11-26T15:23:30, end=2023-11-26T15:26:00) 2023-11-27 13:36:04.738 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20231117 2023-11-27 13:36:04.738 INFO 10 --- [SimpleAsyncTaskExecutor-11,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:04.739 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:04.739 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:04.746 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=44) - no updates (start=2023-11-26T15:26:00, end=2023-11-26T15:28:30) 2023-11-27 13:36:04.747 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=48) - no updates (start=2023-11-26T15:36:00, end=2023-11-26T15:38:30) 2023-11-27 13:36:04.747 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` drop partition (P20231117) 2023-11-27 13:36:04.771 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=45) - no updates (start=2023-11-26T15:28:30, end=2023-11-26T15:31:00) 2023-11-27 13:36:04.792 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=49) - no updates (start=2023-11-26T15:38:30, end=2023-11-26T15:41:00) 2023-11-27 13:36:04.810 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup delay (cluster=darenOB:10, server=3, level=0->1) - 1312 min 2023-11-27 13:36:04.810 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=darenOB:10, server=3, level=1->2, c=0) - fall behind, will continue from 2023-11-26T13:36:00 (source=2023-11-26T15:41:00, target=2023-11-24T10:17:53, granule=600000ms, delay=192187000ms) 2023-11-27 13:36:04.813 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=46) - no updates (start=2023-11-26T15:31:00, end=2023-11-26T15:33:30) 2023-11-27 13:36:04.830 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=49) - no updates (start=2023-11-26T15:38:30, end=2023-11-26T15:41:00) 2023-11-27 13:36:04.842 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:04.842 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:04.854 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063355, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:04.859 INFO 10 --- [SimpleAsyncTaskExecutor-11,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:04.859 INFO 10 --- [SimpleAsyncTaskExecutor-11,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:04.865 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=0) - no updates (start=2023-11-26T13:36:00, end=2023-11-26T13:48:00) 2023-11-27 13:36:04.865 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=0) - no updates (start=2023-11-26T13:36:00, end=2023-11-26T13:48:00) 2023-11-27 13:36:04.866 INFO 10 --- [SimpleAsyncTaskExecutor-11,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select event_id, wait_class as wait_class_name, wait_class_id, event as event_name from v$session_event where sid = (select sid from v$session_event limit 1) 2023-11-27 13:36:04.895 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup delay (cluster=darenOB:10, server=5, level=0->1) - 1312 min 2023-11-27 13:36:04.895 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=darenOB:10, server=5, level=1->2, c=0) - fall behind, will continue from 2023-11-26T13:36:00 (source=2023-11-26T15:41:00, target=2023-11-24T10:17:53, granule=600000ms, delay=192187000ms) 2023-11-27 13:36:04.896 INFO 10 --- [http-nio-0.0.0.0-8080-exec-6,04e6e081c6e64ed5,e9faad245b90] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.11, traceId=04e6e081c6e64ed5, duration=218 ms] 2023-11-27 13:36:04.902 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2023-11-27 13:36:04.936 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20231118 2023-11-27 13:36:04.938 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:04.939 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:04.939 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` drop partition (P20231118) 2023-11-27 13:36:04.942 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=47) - no updates (start=2023-11-26T15:33:30, end=2023-11-26T15:36:00) 2023-11-27 13:36:04.943 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=1) - no updates (start=2023-11-26T13:48:00, end=2023-11-26T14:00:00) 2023-11-27 13:36:04.936 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=0) - no updates (start=2023-11-26T13:36:00, end=2023-11-26T13:48:00) 2023-11-27 13:36:04.950 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063355, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:04.951 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063355, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:04.981 INFO 10 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2023-11-27 13:36:05.025 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=1) - no updates (start=2023-11-26T13:48:00, end=2023-11-26T14:00:00) 2023-11-27 13:36:05.186 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063365, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:05.187 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063365, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:05.187 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063365, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:05.190 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063365, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:05.190 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063365, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:05.235 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=1) - no updates (start=2023-11-26T13:48:00, end=2023-11-26T14:00:00) 2023-11-27 13:36:05.237 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=48) - no updates (start=2023-11-26T15:36:00, end=2023-11-26T15:38:30) 2023-11-27 13:36:05.240 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=133.197.204.3, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=2, status=UNAVAILABLE, innerStatus=INACTIVE, startTime=1970-01-01T00:00Z, stopTime=1970-01-01T00:00Z}, oldStatus=UNAVAILABLE, newStatus=UNAVAILABLE 2023-11-27 13:36:05.244 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=2) - no updates (start=2023-11-26T14:00:00, end=2023-11-26T14:12:00) 2023-11-27 13:36:05.292 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:05.293 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:05.298 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:05.298 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:05.300 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:05.309 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:05.309 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:05.309 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=2) - no updates (start=2023-11-26T14:00:00, end=2023-11-26T14:12:00) 2023-11-27 13:36:05.310 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:05.313 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:05.313 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:05.318 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2023-11-27 13:36:05.360 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=3) - no updates (start=2023-11-26T14:12:00, end=2023-11-26T14:24:00) 2023-11-27 13:36:05.408 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=3) - no updates (start=2023-11-26T14:12:00, end=2023-11-26T14:24:00) 2023-11-27 13:36:05.421 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=darenOB, obClusterId=10 2023-11-27 13:36:05.421 INFO 10 --- [ForkJoinPool-6-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=darenOB, obClusterId=10, vpcId=1, vpcName=system-default, type=PRIMARY, status=UNAVAILABLE, syncStatus=VALID, rootserverJson={"ObCluster":"darenOB","ObClusterId":10,"ObRegion":"darenOB","ObRegionId":10,"ReadonlyRsList":[],"RsList":[{"address":"133.197.204.10:2882","role":"LEADER","sql_port":2881},{"address":"133.197.204.11:2882","role":"FOLLOWER","sql_port":2881},{"address":"133.197.204.12:2882","role":"FOLLOWER","sql_port":2881},{"address":"133.197.204.13:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1700792522216000}, createTime=2022-11-29T15:05:37+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-11-27 13:36:05.455 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=4) - no updates (start=2023-11-26T14:24:00, end=2023-11-26T14:36:00) 2023-11-27 13:36:05.461 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=4) - no updates (start=2023-11-26T14:24:00, end=2023-11-26T14:36:00) 2023-11-27 13:36:05.533 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=49) - no updates (start=2023-11-26T15:38:30, end=2023-11-26T15:41:00) 2023-11-27 13:36:05.546 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup delay (cluster=darenOB:10, server=6, level=0->1) - 1312 min 2023-11-27 13:36:05.546 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=darenOB:10, server=6, level=1->2, c=0) - fall behind, will continue from 2023-11-26T13:36:00 (source=2023-11-26T15:41:00, target=2023-11-24T10:17:53, granule=600000ms, delay=192187000ms) 2023-11-27 13:36:05.552 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=5) - no updates (start=2023-11-26T14:36:00, end=2023-11-26T14:48:00) 2023-11-27 13:36:05.554 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=2) - no updates (start=2023-11-26T14:00:00, end=2023-11-26T14:12:00) 2023-11-27 13:36:05.555 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=5) - no updates (start=2023-11-26T14:36:00, end=2023-11-26T14:48:00) 2023-11-27 13:36:05.559 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=0) - no updates (start=2023-11-26T13:36:00, end=2023-11-26T13:48:00) 2023-11-27 13:36:05.567 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=6) - no updates (start=2023-11-26T14:48:00, end=2023-11-26T15:00:00) 2023-11-27 13:36:05.584 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:05.584 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:05.588 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:05.596 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=6) - no updates (start=2023-11-26T14:48:00, end=2023-11-26T15:00:00) 2023-11-27 13:36:05.602 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=1) - no updates (start=2023-11-26T13:48:00, end=2023-11-26T14:00:00) 2023-11-27 13:36:05.613 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:05.613 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:05.620 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2023-11-27 13:36:05.621 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=3) - no updates (start=2023-11-26T14:12:00, end=2023-11-26T14:24:00) 2023-11-27 13:36:05.621 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=7) - no updates (start=2023-11-26T15:00:00, end=2023-11-26T15:12:00) 2023-11-27 13:36:05.641 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=7) - no updates (start=2023-11-26T15:00:00, end=2023-11-26T15:12:00) 2023-11-27 13:36:05.642 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=2) - no updates (start=2023-11-26T14:00:00, end=2023-11-26T14:12:00) 2023-11-27 13:36:05.679 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=8) - no updates (start=2023-11-26T15:12:00, end=2023-11-26T15:24:00) 2023-11-27 13:36:05.681 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=4) - no updates (start=2023-11-26T14:24:00, end=2023-11-26T14:36:00) 2023-11-27 13:36:05.697 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=3) - no updates (start=2023-11-26T14:12:00, end=2023-11-26T14:24:00) 2023-11-27 13:36:05.698 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20231203 2023-11-27 13:36:05.711 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:05.712 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:05.712 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=8) - no updates (start=2023-11-26T15:12:00, end=2023-11-26T15:24:00) 2023-11-27 13:36:05.751 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=4) - no updates (start=2023-11-26T14:24:00, end=2023-11-26T14:36:00) 2023-11-27 13:36:05.753 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=5) - no updates (start=2023-11-26T14:36:00, end=2023-11-26T14:48:00) 2023-11-27 13:36:05.778 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20231203 values less than (1701648000000000)) 2023-11-27 13:36:05.789 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=9) - no updates (start=2023-11-26T15:24:00, end=2023-11-26T15:36:00) 2023-11-27 13:36:05.791 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=9) - no updates (start=2023-11-26T15:24:00, end=2023-11-26T15:36:00) 2023-11-27 13:36:05.792 INFO 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=darenOB:10, server=3, level=1->2, c=10) - not enough cached interval (source=2023-11-26T15:41:00, target=2023-11-26T15:36:00, granule=600000ms, delay=300000ms) 2023-11-27 13:36:05.795 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=5) - no updates (start=2023-11-26T14:36:00, end=2023-11-26T14:48:00) 2023-11-27 13:36:05.799 INFO 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=darenOB:10, server=4, level=1->2, c=10) - not enough cached interval (source=2023-11-26T15:41:00, target=2023-11-26T15:36:00, granule=600000ms, delay=300000ms) 2023-11-27 13:36:05.835 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=6) - no updates (start=2023-11-26T14:48:00, end=2023-11-26T15:00:00) 2023-11-27 13:36:05.844 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=6) - no updates (start=2023-11-26T14:48:00, end=2023-11-26T15:00:00) 2023-11-27 13:36:05.847 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=7) - no updates (start=2023-11-26T15:00:00, end=2023-11-26T15:12:00) 2023-11-27 13:36:05.867 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=8) - no updates (start=2023-11-26T15:12:00, end=2023-11-26T15:24:00) 2023-11-27 13:36:05.895 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=9) - no updates (start=2023-11-26T15:24:00, end=2023-11-26T15:36:00) 2023-11-27 13:36:05.896 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=7) - no updates (start=2023-11-26T15:00:00, end=2023-11-26T15:12:00) 2023-11-27 13:36:05.906 INFO 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=darenOB:10, server=6, level=1->2, c=10) - not enough cached interval (source=2023-11-26T15:41:00, target=2023-11-26T15:36:00, granule=600000ms, delay=300000ms) 2023-11-27 13:36:05.962 WARN 10 --- [pool-ocp-schedules-3,,] ironmentPostProcessorApplicationListener : Your Okta Issuer URL is missing. You can copy your domain from the Okta Developer Console. Follow these instructions to find it: https://bit.ly/finding-okta-domain To fix this add the `okta.oauth2.issuer` property to your application environments. 2023-11-27 13:36:05.964 INFO 10 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT5.75235054S 2023-11-27 13:36:05.964 INFO 10 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2023-11-27 13:36:05.980 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=8) - no updates (start=2023-11-26T15:12:00, end=2023-11-26T15:24:00) 2023-11-27 13:36:06.026 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=9) - no updates (start=2023-11-26T15:24:00, end=2023-11-26T15:36:00) 2023-11-27 13:36:06.074 INFO 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=darenOB:10, server=5, level=1->2, c=10) - not enough cached interval (source=2023-11-26T15:41:00, target=2023-11-26T15:36:00, granule=600000ms, delay=300000ms) 2023-11-27 13:36:06.075 INFO 10 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=7234ms, period=120000ms 2023-11-27 13:36:06.129 INFO 10 --- [pool-task-coordinator1,,] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-24 10:40:05.0, nextRun=2023-11-24 10:50:04.0, currentDate=Mon Nov 27 13:36:06 CST 2023 2023-11-27 13:36:06.129 INFO 10 --- [pool-task-coordinator1,,] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=1, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2023-11-24 10:40:05.0, nextRunTime=2023-11-24 10:50:04.0} 2023-11-27 13:36:06.156 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:06.157 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:06.195 INFO 10 --- [pool-task-coordinator1,,] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2023-11-27 13:36:06.198 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:06.198 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Partition rollover daemon, taskDefinitionId=1, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:06.202 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2023-11-27 13:36:06.275 INFO 10 --- [pool-ocp-schedules-3,,] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2023-11-27 13:36:06.276 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2023-11-27 13:36:06.291 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20231204 2023-11-27 13:36:06.299 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:06.300 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:06.314 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20231204 values less than (1701734400000000)) 2023-11-27 13:36:06.565 WARN 10 --- [pool-Check host status3,,24738553ce85] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check failed. host=3, ip=133.197.206.92, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.92:62888 [/133.197.206.92] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.getHostTime(AgentExecutor.java:127) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.doCheckHostStatus(HostCheckManagerImpl.java:206) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostStatus(HostCheckManagerImpl.java:168) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.92:62888 [/133.197.206.92] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 24 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 33 common frames omitted 2023-11-27 13:36:06.567 WARN 10 --- [pool-Check host status1,,16cff7425ff1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check failed. host=1, ip=133.197.206.97, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.97:62888 [/133.197.206.97] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.getHostTime(AgentExecutor.java:127) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.doCheckHostStatus(HostCheckManagerImpl.java:206) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostStatus(HostCheckManagerImpl.java:168) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.97:62888 [/133.197.206.97] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 24 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 33 common frames omitted 2023-11-27 13:36:06.568 WARN 10 --- [pool-Check host status2,,daf979afdf7e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check failed. host=2, ip=133.197.204.3, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.204.3:62888 [/133.197.204.3] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.getHostTime(AgentExecutor.java:127) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.doCheckHostStatus(HostCheckManagerImpl.java:206) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostStatus(HostCheckManagerImpl.java:168) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.204.3:62888 [/133.197.204.3] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 24 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 33 common frames omitted 2023-11-27 13:36:06.606 INFO 10 --- [pool-Check host status3,,24738553ce85] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [3:133.197.206.92], from status [AVAILABLE] to [OFFLINE] 2023-11-27 13:36:06.615 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-24 03:00:00.0, nextRun=2023-11-25 03:00:00.0, currentDate=Mon Nov 27 13:36:06 CST 2023 2023-11-27 13:36:06.615 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=2, name=Archive alarm history data, templateName=ArchiveAlarmHistorySchedule, scheduleRule=0 0 3 * * *, lastRunTime=2023-11-24 03:00:00.0, nextRunTime=2023-11-25 03:00:00.0} 2023-11-27 13:36:06.628 INFO 10 --- [pool-Check host status1,,16cff7425ff1] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [1:133.197.206.97], from status [ONLINE] to [OFFLINE] 2023-11-27 13:36:06.629 INFO 10 --- [pool-Check host status1,,16cff7425ff1] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:36:06.607 WARN 10 --- [pool-Check host status8,,904533283ee8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check failed. host=9, ip=133.197.206.88, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.88:62888 [/133.197.206.88] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.getHostTime(AgentExecutor.java:127) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.doCheckHostStatus(HostCheckManagerImpl.java:206) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostStatus(HostCheckManagerImpl.java:168) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.88:62888 [/133.197.206.88] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 24 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 33 common frames omitted 2023-11-27 13:36:06.631 INFO 10 --- [pool-Check host status2,,daf979afdf7e] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [2:133.197.204.3], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:36:06.631 INFO 10 --- [pool-Check host status2,,daf979afdf7e] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:36:06.634 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Archive alarm history data, nodeCount=1, taskCount=1 2023-11-27 13:36:06.634 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:06.634 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Archive alarm history data, taskDefinitionId=2, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:06.638 INFO 10 --- [pool-Check host status3,,24738553ce85] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:36:06.645 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:06.645 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:06.646 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2023-11-27 13:36:06.673 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-24 10:49:07.0, nextRun=2023-11-24 10:50:05.0, currentDate=Mon Nov 27 13:36:06 CST 2023 2023-11-27 13:36:06.674 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-11-24 10:49:07.0, nextRunTime=2023-11-24 10:50:05.0} 2023-11-27 13:36:06.677 INFO 10 --- [pool-Check host status2,,daf979afdf7e] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=2 2023-11-27 13:36:06.677 INFO 10 --- [pool-Check host status2,,daf979afdf7e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2023-11-27 13:36:06.688 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-11-27 13:36:06.688 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:06.688 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:06.695 INFO 10 --- [pool-Check host status8,,904533283ee8] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [9:133.197.206.88], from status [AVAILABLE] to [OFFLINE] 2023-11-27 13:36:06.695 INFO 10 --- [pool-Check host status8,,904533283ee8] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:36:06.722 INFO 10 --- [pool-Check host status2,,daf979afdf7e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:36:06.724 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:36:06.726 INFO 10 --- [pool-Check host status1,,16cff7425ff1] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=1 2023-11-27 13:36:06.726 INFO 10 --- [pool-Check host status1,,16cff7425ff1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2023-11-27 13:36:06.729 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-24 10:49:21.0, nextRun=2023-11-24 10:50:20.0, currentDate=Mon Nov 27 13:36:06 CST 2023 2023-11-27 13:36:06.729 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-11-24 10:49:21.0, nextRunTime=2023-11-24 10:50:20.0} 2023-11-27 13:36:06.731 INFO 10 --- [pool-Check host status3,,24738553ce85] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=3 2023-11-27 13:36:06.732 INFO 10 --- [pool-Check host status3,,24738553ce85] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2023-11-27 13:36:06.736 INFO 10 --- [pool-Check host status3,,24738553ce85] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:36:06.737 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2023-11-27 13:36:06.738 INFO 10 --- [pool-Check host status1,,16cff7425ff1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:36:06.742 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20231205 2023-11-27 13:36:06.747 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:06.747 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:06.748 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-11-27 13:36:06.748 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:06.748 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:06.749 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20231205 values less than (1701820800000000)) 2023-11-27 13:36:06.762 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:36:06.782 INFO 10 --- [pool-Check host status8,,904533283ee8] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=9 2023-11-27 13:36:06.782 INFO 10 --- [pool-Check host status8,,904533283ee8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=9 2023-11-27 13:36:06.807 INFO 10 --- [pool-Check host status8,,904533283ee8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:36:06.825 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-24 10:01:01.0, nextRun=2023-11-24 11:01:00.0, currentDate=Mon Nov 27 13:36:06 CST 2023 2023-11-27 13:36:06.825 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=8, name=Collect cluster snapshot, templateName=CollectClusterSnapshotSchedule, scheduleRule=0 1 * * * *, lastRunTime=2023-11-24 10:01:01.0, nextRunTime=2023-11-24 11:01:00.0} 2023-11-27 13:36:06.872 INFO 10 --- [pool-Check host status3,,24738553ce85] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent status, hostId=3, status=OFFLINE 2023-11-27 13:36:06.925 INFO 10 --- [pool-Check host status1,,16cff7425ff1] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent status, hostId=1, status=OFFLINE 2023-11-27 13:36:06.926 INFO 10 --- [pool-Check host status3,,24738553ce85] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 10 status OFFLINE 2023-11-27 13:36:06.941 INFO 10 --- [pool-Check host status4,,075a09f98e62] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=133.197.204.10, timestamp=2023-11-27T00:36:06.612740310-05:00 2023-11-27 13:36:06.942 INFO 10 --- [pool-Check host status7,,77c9c57b6af2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=7, ip=133.197.204.13, timestamp=2023-11-27T00:36:06.608252949-05:00 2023-11-27 13:36:06.943 INFO 10 --- [pool-Check host status6,,144f4493b486] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=6, ip=133.197.204.12, timestamp=2023-11-27T00:36:06.580081326-05:00 2023-11-27 13:36:06.944 INFO 10 --- [pool-Check host status5,,ee500bda988b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=133.197.204.11, timestamp=2023-11-27T00:36:06.580879055-05:00 2023-11-27 13:36:06.948 INFO 10 --- [pool-Check host status6,,144f4493b486] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=6, lastAvailableTime=2023-11-27T13:36:06.944+08:00 2023-11-27 13:36:06.952 INFO 10 --- [pool-Check host status5,,ee500bda988b] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-11-27T13:36:06.944+08:00 2023-11-27 13:36:06.961 INFO 10 --- [pool-Check host status6,,144f4493b486] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 13 last available time 2023-11-27T13:36:06.944+08:00 and status AVAILABLE 2023-11-27 13:36:06.961 INFO 10 --- [pool-Check host status7,,77c9c57b6af2] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2023-11-27T13:36:06.943+08:00 2023-11-27 13:36:06.963 INFO 10 --- [pool-Check host status4,,075a09f98e62] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-11-27T13:36:06.942+08:00 2023-11-27 13:36:06.971 INFO 10 --- [pool-prometheus-minute-schedule-1,,278d61153042] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1701063420, interval=60 2023-11-27 13:36:06.976 INFO 10 --- [pool-Check host status5,,ee500bda988b] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 12 last available time 2023-11-27T13:36:06.944+08:00 and status AVAILABLE 2023-11-27 13:36:06.977 INFO 10 --- [pool-prometheus-minute-schedule-2,,8f6432f5a740] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1701063420, interval=60 2023-11-27 13:36:06.979 INFO 10 --- [pool-Check host status4,,075a09f98e62] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 11 last available time 2023-11-27T13:36:06.942+08:00 and status AVAILABLE 2023-11-27 13:36:06.984 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect cluster hour metric, nodeCount=2, taskCount=3 2023-11-27 13:36:06.984 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:06.984 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect cluster hour metric, taskDefinitionId=8, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:06.987 INFO 10 --- [pool-Check host status8,,904533283ee8] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent status, hostId=9, status=OFFLINE 2023-11-27 13:36:06.993 INFO 10 --- [pool-Check host status8,,904533283ee8] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 15 status OFFLINE 2023-11-27 13:36:07.015 INFO 10 --- [pool-Check host status1,,16cff7425ff1] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 8 status OFFLINE 2023-11-27 13:36:07.024 INFO 10 --- [pool-prometheus-minute-schedule-2,,8f6432f5a740] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=3, collectTaskCount=3 2023-11-27 13:36:07.024 INFO 10 --- [pool-prometheus-minute-schedule-2,,8f6432f5a740] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1701063366, count=3, elapsed=1699362303658 2023-11-27 13:36:07.037 INFO 10 --- [pool-Check host status7,,77c9c57b6af2] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 14 last available time 2023-11-27T13:36:06.943+08:00 and status AVAILABLE 2023-11-27 13:36:07.106 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/stat, collectAt=1701063366, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:07.148 INFO 10 --- [pool-Check host status4,,075a09f98e62] c.o.o.c.host.service.HostServiceImpl : begin to compare and update status, host id: 4, new status: ONLINE, old status: OFFLINE 2023-11-27 13:36:07.150 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/stat, collectAt=1701063366, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:07.151 INFO 10 --- [pool-Check host status5,,ee500bda988b] c.o.o.c.host.service.HostServiceImpl : begin to compare and update status, host id: 5, new status: ONLINE, old status: OFFLINE 2023-11-27 13:36:07.154 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-24 10:49:26.0, nextRun=2023-11-24 10:50:25.0, currentDate=Mon Nov 27 13:36:07 CST 2023 2023-11-27 13:36:07.155 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-11-24 10:49:26.0, nextRunTime=2023-11-24 10:50:25.0} 2023-11-27 13:36:07.157 INFO 10 --- [pool-Check host status5,,ee500bda988b] c.o.o.c.host.service.HostServiceImpl : update host status success, host id: 5, new status: ONLINE, old status: OFFLINE 2023-11-27 13:36:07.161 INFO 10 --- [pool-Check host status6,,144f4493b486] c.o.o.c.host.service.HostServiceImpl : begin to compare and update status, host id: 6, new status: ONLINE, old status: OFFLINE 2023-11-27 13:36:07.164 INFO 10 --- [pool-Check host status7,,77c9c57b6af2] c.o.o.c.host.service.HostServiceImpl : begin to compare and update status, host id: 7, new status: ONLINE, old status: OFFLINE 2023-11-27 13:36:07.165 INFO 10 --- [pool-Check host status6,,144f4493b486] c.o.o.c.host.service.HostServiceImpl : update host status success, host id: 6, new status: ONLINE, old status: OFFLINE 2023-11-27 13:36:07.180 INFO 10 --- [pool-Check host status4,,075a09f98e62] c.o.o.c.host.service.HostServiceImpl : update host status success, host id: 4, new status: ONLINE, old status: OFFLINE 2023-11-27 13:36:07.200 INFO 10 --- [pool-Check host status6,,144f4493b486] c.o.o.c.host.service.HostServiceImpl : begin to compare and update status, host id: 6, new status: ONLINE, old status: OFFLINE 2023-11-27 13:36:07.209 WARN 10 --- [pool-Check host status6,,144f4493b486] c.o.o.c.host.service.HostServiceImpl : update host status failed, host status already bean updated, host id: 6, new status: ONLINE, old status: OFFLINE 2023-11-27 13:36:07.212 INFO 10 --- [pool-Check host status4,,075a09f98e62] c.o.o.c.host.service.HostServiceImpl : begin to compare and update status, host id: 4, new status: ONLINE, old status: OFFLINE 2023-11-27 13:36:07.214 INFO 10 --- [pool-prometheus-minute-schedule-1,,278d61153042] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=4, collectTaskCount=4 2023-11-27 13:36:07.214 INFO 10 --- [pool-prometheus-minute-schedule-1,,278d61153042] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1701063366, count=4, elapsed=1699362303848 2023-11-27 13:36:07.215 INFO 10 --- [pool-Check host status7,,77c9c57b6af2] c.o.o.c.host.service.HostServiceImpl : update host status success, host id: 7, new status: ONLINE, old status: OFFLINE 2023-11-27 13:36:07.220 INFO 10 --- [pool-Check host status5,,ee500bda988b] c.o.o.c.host.service.HostServiceImpl : begin to compare and update status, host id: 5, new status: ONLINE, old status: OFFLINE 2023-11-27 13:36:07.247 INFO 10 --- [pool-Check host status6,,144f4493b486] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=6, lastAvailableTime=2023-11-27T13:36:07.219+08:00 2023-11-27 13:36:07.255 INFO 10 --- [pool-Check host status7,,77c9c57b6af2] c.o.o.c.host.service.HostServiceImpl : begin to compare and update status, host id: 7, new status: ONLINE, old status: OFFLINE 2023-11-27 13:36:07.263 INFO 10 --- [pool-Check host status6,,144f4493b486] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 13 last available time 2023-11-27T13:36:07.219+08:00 and status AVAILABLE 2023-11-27 13:36:07.267 WARN 10 --- [pool-Check host status5,,ee500bda988b] c.o.o.c.host.service.HostServiceImpl : update host status failed, host status already bean updated, host id: 5, new status: ONLINE, old status: OFFLINE 2023-11-27 13:36:07.269 WARN 10 --- [pool-Check host status7,,77c9c57b6af2] c.o.o.c.host.service.HostServiceImpl : update host status failed, host status already bean updated, host id: 7, new status: ONLINE, old status: OFFLINE 2023-11-27 13:36:07.273 INFO 10 --- [pool-Check host status7,,77c9c57b6af2] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2023-11-27T13:36:07.272+08:00 2023-11-27 13:36:07.273 WARN 10 --- [pool-Check host status4,,075a09f98e62] c.o.o.c.host.service.HostServiceImpl : update host status failed, host status already bean updated, host id: 4, new status: ONLINE, old status: OFFLINE 2023-11-27 13:36:07.290 INFO 10 --- [pool-Check host status7,,77c9c57b6af2] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 14 last available time 2023-11-27T13:36:07.272+08:00 and status AVAILABLE 2023-11-27 13:36:07.297 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62888/metrics/stat, collectAt=1701063366, message=java.net.ConnectException: Connection refused: /133.197.206.92:62888, rootCause=ConnectException: Connection refused 2023-11-27 13:36:07.310 INFO 10 --- [pool-Check host status6,,144f4493b486] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=6 2023-11-27 13:36:07.313 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:07.313 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:07.321 INFO 10 --- [pool-Check host status4,,075a09f98e62] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-11-27T13:36:07.320+08:00 2023-11-27 13:36:07.326 INFO 10 --- [pool-Check host status4,,075a09f98e62] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 11 last available time 2023-11-27T13:36:07.320+08:00 and status AVAILABLE 2023-11-27 13:36:07.341 INFO 10 --- [pool-Check host status5,,ee500bda988b] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-11-27T13:36:07.332+08:00 2023-11-27 13:36:07.343 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2023-11-27 13:36:07.346 INFO 10 --- [pool-Check host status4,,075a09f98e62] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-11-27 13:36:07.351 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-11-27 13:36:07.352 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:07.352 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:07.353 INFO 10 --- [pool-Check host status5,,ee500bda988b] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 12 last available time 2023-11-27T13:36:07.332+08:00 and status AVAILABLE 2023-11-27 13:36:07.361 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62888/metrics/stat, collectAt=1701063366, message=java.net.ConnectException: Connection refused: /133.197.206.97:62888, rootCause=ConnectException: Connection refused 2023-11-27 13:36:07.386 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2023-11-27 13:36:07.388 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20231025 2023-11-27 13:36:07.389 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:07.390 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:07.391 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` drop partition (P20231025) 2023-11-27 13:36:07.397 INFO 10 --- [pool-Check host status7,,77c9c57b6af2] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=7 2023-11-27 13:36:07.428 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-24 10:40:00.0, nextRun=2023-11-24 10:50:00.0, currentDate=Mon Nov 27 13:36:07 CST 2023 2023-11-27 13:36:07.428 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=12, name=Refresh agent config, templateName=RefreshAgentConfigSchedule, scheduleRule=0 */10 * * * *, lastRunTime=2023-11-24 10:40:00.0, nextRunTime=2023-11-24 10:50:00.0} 2023-11-27 13:36:07.433 INFO 10 --- [pool-Check host status5,,ee500bda988b] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-11-27 13:36:07.437 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62888/metrics/stat, collectAt=1701063366, message=java.net.ConnectException: Connection refused: /133.197.206.88:62888, rootCause=ConnectException: Connection refused 2023-11-27 13:36:07.480 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Refresh agent config, nodeCount=1, taskCount=1 2023-11-27 13:36:07.480 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:07.480 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Refresh agent config, taskDefinitionId=12, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:07.486 INFO 10 --- [pool-Check host status7,,77c9c57b6af2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2023-11-27 13:36:07.491 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/extra, collectAt=1701063366, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:07.492 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/stat, collectAt=1701063366, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:07.506 INFO 10 --- [pool-Check host status4,,075a09f98e62] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-11-27 13:36:07.525 INFO 10 --- [pool-ocp-schedules-3,,] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}, BootstrapPropertySource {name='bootstrapProperties-ocp-composite'}, BootstrapPropertySource {name='bootstrapProperties-application-composite'}, BootstrapPropertySource {name='bootstrapProperties-application-default'}] 2023-11-27 13:36:07.540 INFO 10 --- [pool-Check host status5,,ee500bda988b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-11-27 13:36:07.581 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-24 10:48:00.0, nextRun=2023-11-24 10:50:00.0, currentDate=Mon Nov 27 13:36:07 CST 2023 2023-11-27 13:36:07.581 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=14, name=Sync root service list, templateName=SyncRsListSchedule, scheduleRule=0 */2 * * * ?, lastRunTime=2023-11-24 10:48:00.0, nextRunTime=2023-11-24 10:50:00.0} 2023-11-27 13:36:07.585 INFO 10 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2023-11-27 13:36:07.626 INFO 10 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2023-11-27 13:36:07.626 INFO 10 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : set system property ocp.version to 4.0.3 2023-11-27 13:36:07.626 INFO 10 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : set systemProperties as the first PropertySource 2023-11-27 13:36:07.626 INFO 10 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.0.3 2023-11-27 13:36:07.626 INFO 10 --- [pool-ocp-schedules-3,,] o.s.boot.SpringApplication : The following 2 profiles are active: "composite", "default" 2023-11-27 13:36:07.636 INFO 10 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2023-11-27 13:36:07.656 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync root service list, nodeCount=1, taskCount=2 2023-11-27 13:36:07.656 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:07.656 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:07.656 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync root service list, taskDefinitionId=14, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:07.666 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:07.666 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:07.674 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2023-11-27 13:36:07.722 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20231026 2023-11-27 13:36:07.749 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:07.749 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:07.752 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-24 10:40:01.0, nextRun=2023-11-24 10:50:00.0, currentDate=Mon Nov 27 13:36:07 CST 2023 2023-11-27 13:36:07.752 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=15, name=Sync system event, templateName=SyncSystemEventSchedule, scheduleRule=0 */10 * * * *, lastRunTime=2023-11-24 10:40:01.0, nextRunTime=2023-11-24 10:50:00.0} 2023-11-27 13:36:07.758 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` drop partition (P20231026) 2023-11-27 13:36:07.797 INFO 10 --- [pool-Check host status6,,144f4493b486] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2023-11-27 13:36:07.806 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync system event, nodeCount=2, taskCount=3 2023-11-27 13:36:07.806 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:07.806 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync system event, taskDefinitionId=15, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:07.927 INFO 10 --- [pool-ocp-schedules-3,,] o.s.boot.SpringApplication : Started application in 7.765 seconds (JVM running for 184.065) 2023-11-27 13:36:07.927 INFO 10 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT7.766332118S 2023-11-27 13:36:07.931 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-24 10:49:31.0, nextRun=2023-11-24 10:50:30.0, currentDate=Mon Nov 27 13:36:07 CST 2023 2023-11-27 13:36:07.931 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-11-24 10:49:31.0, nextRunTime=2023-11-24 10:50:30.0} 2023-11-27 13:36:07.965 INFO 10 --- [pool-Check host status4,,075a09f98e62] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-11-27 13:36:08.030 INFO 10 --- [pool-Check host status7,,77c9c57b6af2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-11-27 13:36:08.058 INFO 10 --- [pool-Check host status4,,075a09f98e62] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=31 2023-11-27 13:36:08.072 INFO 10 --- [pool-Check host status6,,144f4493b486] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-11-27 13:36:08.088 INFO 10 --- [SimpleAsyncTaskExecutor-11,,] c.o.o.p.a.i.SessionEventSyncService : Sync session wait event info finished, cluster ids is [2, 1] 2023-11-27 13:36:08.095 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000008, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:08.113 INFO 10 --- [pool-Check host status7,,77c9c57b6af2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2023-11-27 13:36:08.122 INFO 10 --- [pool-Check host status6,,144f4493b486] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=37 2023-11-27 13:36:08.154 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-11-27 13:36:08.154 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:08.154 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:08.154 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:08.217 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 8225441, operation:EXECUTE, state: FAILED 2023-11-27 13:36:08.225 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:08.225 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:08.228 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000008, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-11-27T13:36:08.044+08:00, endTime=null} 2023-11-27 13:36:08.235 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2023-11-27 13:36:08.244 INFO 10 --- [pool-Check host status5,,ee500bda988b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-11-27 13:36:08.267 INFO 10 --- [pool-Check host status6,,144f4493b486] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=37 2023-11-27 13:36:08.268 INFO 10 --- [pool-Check host status4,,075a09f98e62] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=31 2023-11-27 13:36:08.277 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20231027 2023-11-27 13:36:08.294 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:08.294 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:08.303 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` drop partition (P20231027) 2023-11-27 13:36:08.303 INFO 10 --- [pool-Check host status7,,77c9c57b6af2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2023-11-27 13:36:08.309 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-24 00:00:01.0, nextRun=2023-11-25 00:00:00.0, currentDate=Mon Nov 27 13:36:08 CST 2023 2023-11-27 13:36:08.309 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=18, name=Sync oceanbase wait event info, templateName=ObWaitEventSyncSchedule, scheduleRule=0 0 0 * * *, lastRunTime=2023-11-24 00:00:01.0, nextRunTime=2023-11-25 00:00:00.0} 2023-11-27 13:36:08.314 WARN 10 --- [pool-subtask-executor1,,] c.o.o.c.t.e.c.w.s.TimeoutSubtaskWorker : Subtask timeout, name=Sync all tenant information, executedSeconds=269196, timeout=300 2023-11-27 13:36:08.338 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync oceanbase wait event info, nodeCount=1, taskCount=1 2023-11-27 13:36:08.338 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:08.338 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync oceanbase wait event info, taskDefinitionId=18, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:08.348 INFO 10 --- [pool-Check host status5,,ee500bda988b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=34 2023-11-27 13:36:08.439 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:08.439 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:08.444 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2023-11-27 13:36:08.449 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.120.0.1:8080 2023-11-27 13:36:08.452 INFO 10 --- [pool-Check host status5,,ee500bda988b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=34 2023-11-27 13:36:08.491 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-24 02:00:02.0, nextRun=2023-11-25 02:00:00.0, currentDate=Mon Nov 27 13:36:08 CST 2023 2023-11-27 13:36:08.491 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=19, name=Clean expired health check record, templateName=CleanExpiredHealthCheckRecordSchedule, scheduleRule=0 0 2 * * ?, lastRunTime=2023-11-24 02:00:02.0, nextRunTime=2023-11-25 02:00:00.0} 2023-11-27 13:36:08.501 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 8225440, operation:EXECUTE, state: FAILED 2023-11-27 13:36:08.512 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000009, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:08.513 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Clean expired health check record, nodeCount=1, taskCount=1 2023-11-27 13:36:08.513 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:08.513 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Clean expired health check record, taskDefinitionId=19, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:08.522 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:08.522 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:08.544 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:08.548 INFO 10 --- [pool-ocp-schedules-3,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] driver class=com.oceanbase.jdbc.Driver 2023-11-27 13:36:08.548 INFO 10 --- [pool-ocp-schedules-3,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] username=meta_user@meta_tenant#obcluster 2023-11-27 13:36:08.548 INFO 10 --- [pool-ocp-schedules-3,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] jdbc url=jdbc:oceanbase://133.197.206.97:2883/meta_database?useUnicode=true&characterEncoding=UTF8&connectTimeout=2000&socketTimeout=30000&rewriteBatchedStatements=true 2023-11-27 13:36:08.548 INFO 10 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.DataSourceInterceptor : dataSource metaDataSource already initialized. skipped. class: com.oceanbase.ocp.server.config.wrapper.MetadbDataSourceWrapper 2023-11-27 13:36:08.557 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000009, name=Refresh all agent config, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.RefreshAllAgentConfigTask, seriesId=0, startTime=2023-11-27T13:36:08.502+08:00, endTime=null} 2023-11-27 13:36:08.578 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:08.578 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:08.590 WARN 10 --- [pool-subtask-executor3,,] c.o.o.c.t.e.c.w.s.TimeoutSubtaskWorker : Subtask timeout, name=Sync all tenant information, executedSeconds=269196, timeout=300 2023-11-27 13:36:08.617 INFO 10 --- [pool-Check host status6,,144f4493b486] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-11-27 13:36:08.594 INFO 10 --- [pool-Check host status7,,77c9c57b6af2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-11-27 13:36:08.598 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-24 10:45:02.0, nextRun=2023-11-24 10:50:00.0, currentDate=Mon Nov 27 13:36:08 CST 2023 2023-11-27 13:36:08.621 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=20, name=Check all cluster configUrl, templateName=CheckClusterConfigUrlSchedule, scheduleRule=0 */5 * * * ?, lastRunTime=2023-11-24 10:45:02.0, nextRunTime=2023-11-24 10:50:00.0} 2023-11-27 13:36:08.600 INFO 10 --- [pool-Check host status4,,075a09f98e62] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-11-27 13:36:08.601 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2023-11-27 13:36:08.601 INFO 10 --- [pool-Check host status5,,ee500bda988b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-11-27 13:36:08.652 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000008, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000006, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:08.042+08:00, sub_task_instance_id=10000008}, listMap={host_ids=[1, 2, 3, 4, 5, 6, 7, 9]}}, executor=10.120.0.1 2023-11-27 13:36:08.658 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 8225442, operation:EXECUTE, state: FAILED 2023-11-27 13:36:08.659 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000010, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:08.665 INFO 10 --- [pool-Check host status6,,144f4493b486] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=38 2023-11-27 13:36:08.666 INFO 10 --- [pool-Check host status4,,075a09f98e62] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=32 2023-11-27 13:36:08.674 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20231203 2023-11-27 13:36:08.685 INFO 10 --- [pool-Check host status5,,ee500bda988b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=35 2023-11-27 13:36:08.685 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=CheckClusterConfigUrlSchedule, nodeCount=1, taskCount=1 2023-11-27 13:36:08.685 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:08.685 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=CheckClusterConfigUrlSchedule, taskDefinitionId=20, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:08.686 INFO 10 --- [pool-Check host status7,,77c9c57b6af2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2023-11-27 13:36:08.686 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:08.686 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:08.693 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20231203 values less than (1701648000000000)) 2023-11-27 13:36:08.755 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000010, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=1, startTime=2023-11-27T13:36:08.649+08:00, endTime=null} 2023-11-27 13:36:08.758 WARN 10 --- [pool-subtask-executor5,,] c.o.o.c.t.e.c.w.s.TimeoutSubtaskWorker : Subtask timeout, name=Collect all tenants mini compactions, executedSeconds=269196, timeout=600 2023-11-27 13:36:08.804 INFO 10 --- [pool-Check host status6,,144f4493b486] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=38 2023-11-27 13:36:08.820 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-24 10:45:02.0, nextRun=2023-11-24 10:50:00.0, currentDate=Mon Nov 27 13:36:08 CST 2023 2023-11-27 13:36:08.820 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=21, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2023-11-24 10:45:02.0, nextRunTime=2023-11-24 10:50:00.0} 2023-11-27 13:36:08.890 INFO 10 --- [pool-Check host status4,,075a09f98e62] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=32 2023-11-27 13:36:08.991 INFO 10 --- [pool-Check host status5,,ee500bda988b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=35 2023-11-27 13:36:08.992 INFO 10 --- [pool-Check host status7,,77c9c57b6af2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2023-11-27 13:36:09.001 INFO 10 --- [pool-subtask-executor4,554481c3fd1b426b,a8a0ae9d6cc0] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000009, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000007, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:08.500+08:00, sub_task_instance_id=10000009}, listMap={}}, executor=10.120.0.1 2023-11-27 13:36:09.111 INFO 10 --- [pool-subtask-executor4,554481c3fd1b426b,a8a0ae9d6cc0] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=6 2023-11-27 13:36:09.111 INFO 10 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=7 2023-11-27 13:36:09.121 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000011, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:09.123 INFO 10 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=5 2023-11-27 13:36:09.134 INFO 10 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=4 2023-11-27 13:36:09.211 INFO 10 --- [pool-Check host status6,,144f4493b486] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-11-27 13:36:09.250 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000011, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=0, startTime=2023-11-27T13:36:09.110+08:00, endTime=null} 2023-11-27 13:36:09.276 INFO 10 --- [pool-Check host status6,,144f4493b486] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=39 2023-11-27 13:36:09.277 INFO 10 --- [pool-ocp-schedules-3,,] c.o.ocp.core.property.PropertyService : Refresh config properties done 2023-11-27 13:36:09.295 INFO 10 --- [pool-Check host status7,,77c9c57b6af2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-11-27 13:36:09.298 INFO 10 --- [pool-Check host status4,,075a09f98e62] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-11-27 13:36:09.317 INFO 10 --- [pool-subtask-executor6,554481c3fd1b426b,4d777ffc0273] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000010, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=8000008, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:08.648+08:00, sub_task_instance_id=10000010}, listMap={cluster_ids=[1, 2]}}, executor=10.120.0.1 2023-11-27 13:36:09.336 INFO 10 --- [pool-Check host status4,,075a09f98e62] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=33 2023-11-27 13:36:09.349 INFO 10 --- [pool-Check host status5,,ee500bda988b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-11-27 13:36:09.364 INFO 10 --- [pool-Check host status7,,77c9c57b6af2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2023-11-27 13:36:09.393 INFO 10 --- [pool-Check host status6,,144f4493b486] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=39 2023-11-27 13:36:09.403 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:09.403 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:09.406 INFO 10 --- [pool-Check host status5,,ee500bda988b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=36 2023-11-27 13:36:09.411 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2023-11-27 13:36:09.497 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2023-11-27 13:36:09.499 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000001, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:09.514 INFO 10 --- [pool-Check host status4,,075a09f98e62] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=33 2023-11-27 13:36:09.516 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20231204 2023-11-27 13:36:09.526 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:09.526 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:09.529 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20231204 values less than (1701734400000000)) 2023-11-27 13:36:09.530 INFO 10 --- [pool-Check host status7,,77c9c57b6af2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2023-11-27 13:36:09.540 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000001, name=Partition rollover daemon, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=2023-11-27T13:36:09.490+08:00, endTime=null} 2023-11-27 13:36:09.545 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270794, maxInactiveSeconds=180 2023-11-27 13:36:09.545 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270794, maxInactiveSeconds=180 2023-11-27 13:36:09.545 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270794, maxInactiveSeconds=180 2023-11-27 13:36:09.545 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270794, maxInactiveSeconds=180 2023-11-27 13:36:09.545 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270794, maxInactiveSeconds=180 2023-11-27 13:36:09.545 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270794, maxInactiveSeconds=180 2023-11-27 13:36:09.545 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270794, maxInactiveSeconds=180 2023-11-27 13:36:09.545 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270794, maxInactiveSeconds=180 2023-11-27 13:36:09.545 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270434, maxInactiveSeconds=180 2023-11-27 13:36:09.545 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270434, maxInactiveSeconds=180 2023-11-27 13:36:09.549 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270434, maxInactiveSeconds=180 2023-11-27 13:36:09.549 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270434, maxInactiveSeconds=180 2023-11-27 13:36:09.549 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270434, maxInactiveSeconds=180 2023-11-27 13:36:09.549 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270434, maxInactiveSeconds=180 2023-11-27 13:36:09.549 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270434, maxInactiveSeconds=180 2023-11-27 13:36:09.550 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270434, maxInactiveSeconds=180 2023-11-27 13:36:09.550 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270434, maxInactiveSeconds=180 2023-11-27 13:36:09.550 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:56, inactiveSeconds=270433, maxInactiveSeconds=180 2023-11-27 13:36:09.550 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:56, inactiveSeconds=270433, maxInactiveSeconds=180 2023-11-27 13:36:09.550 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270434, maxInactiveSeconds=180 2023-11-27 13:36:09.550 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270434, maxInactiveSeconds=180 2023-11-27 13:36:09.550 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270434, maxInactiveSeconds=180 2023-11-27 13:36:09.550 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270434, maxInactiveSeconds=180 2023-11-27 13:36:09.550 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270434, maxInactiveSeconds=180 2023-11-27 13:36:09.550 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270434, maxInactiveSeconds=180 2023-11-27 13:36:09.550 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270434, maxInactiveSeconds=180 2023-11-27 13:36:09.550 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270434, maxInactiveSeconds=180 2023-11-27 13:36:09.550 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270434, maxInactiveSeconds=180 2023-11-27 13:36:09.550 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270434, maxInactiveSeconds=180 2023-11-27 13:36:09.551 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270434, maxInactiveSeconds=180 2023-11-27 13:36:09.551 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270434, maxInactiveSeconds=180 2023-11-27 13:36:09.551 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270434, maxInactiveSeconds=180 2023-11-27 13:36:09.551 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270434, maxInactiveSeconds=180 2023-11-27 13:36:09.551 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270434, maxInactiveSeconds=180 2023-11-27 13:36:09.551 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270434, maxInactiveSeconds=180 2023-11-27 13:36:09.551 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270434, maxInactiveSeconds=180 2023-11-27 13:36:09.551 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270434, maxInactiveSeconds=180 2023-11-27 13:36:09.551 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270434, maxInactiveSeconds=180 2023-11-27 13:36:09.551 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270434, maxInactiveSeconds=180 2023-11-27 13:36:09.551 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270434, maxInactiveSeconds=180 2023-11-27 13:36:09.551 WARN 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Host [id=1, ip=133.197.206.97, status=OFFLINE] not online, skip check exporter 2023-11-27 13:36:09.551 WARN 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Host [id=2, ip=133.197.204.3, status=OFFLINE] not online, skip check exporter 2023-11-27 13:36:09.551 WARN 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Host [id=3, ip=133.197.206.92, status=OFFLINE] not online, skip check exporter 2023-11-27 13:36:09.563 INFO 10 --- [pool-Check host status5,,ee500bda988b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=36 2023-11-27 13:36:09.591 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=133.197.204.10] desired exporter count: 8, actual exporter count: 0 2023-11-27 13:36:09.592 WARN 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Send exporter not enough alarm, ip=133.197.204.10, deployedServices=[OB_CLUSTER, OB_PROXY], expectedCount=8, activeCount=0 2023-11-27 13:36:09.625 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2023-11-27 13:36:09.625 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:09.626 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=SqlParseTaskSchedule, taskDefinitionId=21, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:09.649 INFO 10 --- [pool-subtask-executor7,554481c3fd1b426b,8e91b7aa7ab6] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000011, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=8000008, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:09.108+08:00, sub_task_instance_id=10000011}, listMap={cluster_ids=[1, 2]}}, executor=10.120.0.1 2023-11-27 13:36:09.679 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.a.service.OcpAlarmServiceImpl : delete Done, affectRow=1, id=25983 2023-11-27 13:36:09.693 INFO 10 --- [pool-task-coordinator2,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Mark failed task, entity=TaskInstanceEntity{id=6670995, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:09.700 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-24 10:49:00.0, nextRun=2023-11-24 10:50:00.0, currentDate=Mon Nov 27 13:36:09 CST 2023 2023-11-27 13:36:09.700 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-24 10:49:00.0, nextRunTime=2023-11-24 10:50:00.0} 2023-11-27 13:36:09.701 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000002, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:09.756 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000002, name=Archive alarm history data, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.alarm.ArchiveAlarmHistoryDataTask, seriesId=0, startTime=2023-11-27T13:36:09.680+08:00, endTime=null} 2023-11-27 13:36:09.762 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:36:09.769 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:36:09.817 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.a.service.OcpAlarmServiceImpl : create alarm event Done, affectRow=1, id=2000001 2023-11-27 13:36:09.818 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-11-27 13:36:09.826 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:09.826 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:09.856 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:09.856 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:09.890 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2023-11-27 13:36:09.946 INFO 10 --- [pool-subtask-executor8,554481c3fd1b426b,7a5a3ce68651] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000001, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000001, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:09.464+08:00, sub_task_instance_id=10000001}, listMap={}}, executor=10.120.0.1 2023-11-27 13:36:09.947 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:6670995, status:FAILED 2023-11-27 13:36:09.973 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-24 10:49:00.0, nextRun=2023-11-24 10:50:00.0, currentDate=Mon Nov 27 13:36:09 CST 2023 2023-11-27 13:36:09.973 INFO 10 --- [ForkJoinPool.commonPool-worker-2,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 4 agent sql collect property 2023-11-27 13:36:09.974 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-24 10:49:00.0, nextRunTime=2023-11-24 10:50:00.0} 2023-11-27 13:36:09.975 INFO 10 --- [pool-subtask-executor7,554481c3fd1b426b,8e91b7aa7ab6] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000011, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:09.976 INFO 10 --- [pool-Check host status9,,c0d7583006ad] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=133.197.206.97, originalStatus=ONLINE 2023-11-27 13:36:09.985 INFO 10 --- [pool-Check host status10,,4382b94d5e22] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=133.197.204.3, originalStatus=OFFLINE 2023-11-27 13:36:09.998 INFO 10 --- [pool-Check host status11,,d672cf4ce3c1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=133.197.206.92, originalStatus=AVAILABLE 2023-11-27 13:36:10.006 INFO 10 --- [pool-Check host status12,,1b34771788d8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=133.197.204.10, originalStatus=OFFLINE 2023-11-27 13:36:10.010 INFO 10 --- [pool-Check host status13,,8714d1c389f5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=133.197.204.11, originalStatus=OFFLINE 2023-11-27 13:36:10.015 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000003, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:10.022 INFO 10 --- [pool-Check host status14,,ae847bb07967] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=6, ip=133.197.204.12, originalStatus=OFFLINE 2023-11-27 13:36:10.044 INFO 10 --- [pool-Check host status12,,1b34771788d8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-11-27 13:36:10.047 INFO 10 --- [pool-Check host status15,,da4a0c4b6da0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=133.197.204.13, originalStatus=OFFLINE 2023-11-27 13:36:10.047 INFO 10 --- [pool-Check host status16,,caf7ad2317c3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=9, ip=133.197.206.88, originalStatus=AVAILABLE 2023-11-27 13:36:10.054 INFO 10 --- [ForkJoinPool.commonPool-worker-0,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 5 agent sql collect property 2023-11-27 13:36:10.055 INFO 10 --- [pool-subtask-executor4,554481c3fd1b426b,a8a0ae9d6cc0] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 6 agent sql collect property 2023-11-27 13:36:10.057 INFO 10 --- [pool-Check host status13,,8714d1c389f5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-11-27 13:36:10.070 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2023-11-27 13:36:10.074 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20231205 2023-11-27 13:36:10.074 INFO 10 --- [pool-subtask-executor6,554481c3fd1b426b,4d777ffc0273] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:10.078 INFO 10 --- [ForkJoinPool.commonPool-worker-1,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 7 agent sql collect property 2023-11-27 13:36:10.078 INFO 10 --- [pool-Check host status10,,4382b94d5e22] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:36:10.080 INFO 10 --- [pool-subtask-executor6,554481c3fd1b426b,4d777ffc0273] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:10.080 INFO 10 --- [pool-subtask-executor6,554481c3fd1b426b,4d777ffc0273] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:10.082 INFO 10 --- [pool-subtask-executor6,554481c3fd1b426b,4d777ffc0273] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:10.074 INFO 10 --- [pool-Check host status11,,d672cf4ce3c1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:36:10.089 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:6670995, status:FAILED 2023-11-27 13:36:10.096 INFO 10 --- [pool-subtask-executor6,554481c3fd1b426b,4d777ffc0273] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:10.096 INFO 10 --- [pool-subtask-executor6,554481c3fd1b426b,4d777ffc0273] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:10.097 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-11-27 13:36:10.097 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:10.097 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:10.097 INFO 10 --- [pool-subtask-executor6,554481c3fd1b426b,4d777ffc0273] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2023-11-27 13:36:10.099 INFO 10 --- [pool-Check host status9,,c0d7583006ad] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:36:10.112 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:10.112 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:10.129 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.a.service.OcpAlarmServiceImpl : create alarm event history Done, affectRow=1, id=2000001 2023-11-27 13:36:10.145 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20231205 values less than (1701820800000000)) 2023-11-27 13:36:10.174 INFO 10 --- [pool-Check host status15,,da4a0c4b6da0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2023-11-27 13:36:10.175 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000003, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-11-27T13:36:09.984+08:00, endTime=null} 2023-11-27 13:36:10.177 INFO 10 --- [pool-Check host status14,,ae847bb07967] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2023-11-27 13:36:10.207 INFO 10 --- [pool-subtask-executor6,554481c3fd1b426b,4d777ffc0273] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:10.207 INFO 10 --- [pool-subtask-executor6,554481c3fd1b426b,4d777ffc0273] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:10.210 INFO 10 --- [pool-task-coordinator2,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Mark failed task, entity=TaskInstanceEntity{id=6670996, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:10.212 INFO 10 --- [pool-subtask-executor6,554481c3fd1b426b,4d777ffc0273] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [133.197.204.10, 2882] 2023-11-27 13:36:10.219 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-24 10:49:00.0, nextRun=2023-11-24 10:50:00.0, currentDate=Mon Nov 27 13:36:10 CST 2023 2023-11-27 13:36:10.219 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-24 10:49:00.0, nextRunTime=2023-11-24 10:50:00.0} 2023-11-27 13:36:10.227 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063370, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:10.228 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063370, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:10.229 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063370, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:10.232 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063370, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:10.232 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063370, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:10.239 INFO 10 --- [pool-subtask-executor6,554481c3fd1b426b,4d777ffc0273] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:10.239 INFO 10 --- [pool-subtask-executor6,554481c3fd1b426b,4d777ffc0273] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:10.245 INFO 10 --- [pool-subtask-executor6,554481c3fd1b426b,4d777ffc0273] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [133.197.204.11, 2882] 2023-11-27 13:36:10.263 INFO 10 --- [pool-Check host status16,,caf7ad2317c3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:36:10.266 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=133.197.204.11] desired exporter count: 8, actual exporter count: 0 2023-11-27 13:36:10.266 WARN 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Send exporter not enough alarm, ip=133.197.204.11, deployedServices=[OB_CLUSTER, OB_PROXY], expectedCount=8, activeCount=0 2023-11-27 13:36:10.271 INFO 10 --- [pool-subtask-executor6,554481c3fd1b426b,4d777ffc0273] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:10.272 INFO 10 --- [pool-subtask-executor6,554481c3fd1b426b,4d777ffc0273] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:10.275 INFO 10 --- [pool-subtask-executor6,554481c3fd1b426b,4d777ffc0273] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [133.197.204.12, 2882] 2023-11-27 13:36:10.286 INFO 10 --- [pool-subtask-executor6,554481c3fd1b426b,4d777ffc0273] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:10.286 INFO 10 --- [pool-subtask-executor6,554481c3fd1b426b,4d777ffc0273] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:10.291 INFO 10 --- [pool-subtask-executor6,554481c3fd1b426b,4d777ffc0273] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [133.197.204.13, 2882] 2023-11-27 13:36:10.322 INFO 10 --- [pool-subtask-executor6,554481c3fd1b426b,4d777ffc0273] c.o.o.s.t.b.cluster.UpdateRsListTask : assembled rootserver_json={"ObCluster":"darenOB","ObClusterId":10,"ObRegion":"darenOB","ObRegionId":10,"ReadonlyRsList":[],"RsList":[{"address":"133.197.204.10:2882","role":"LEADER","sql_port":2881},{"address":"133.197.204.11:2882","role":"FOLLOWER","sql_port":2881},{"address":"133.197.204.12:2882","role":"FOLLOWER","sql_port":2881},{"address":"133.197.204.13:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1701063370316000} 2023-11-27 13:36:10.327 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-11-27 13:36:10.327 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:10.327 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:10.359 INFO 10 --- [pool-subtask-executor4,554481c3fd1b426b,a8a0ae9d6cc0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2023-11-27 13:36:10.365 INFO 10 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-11-27 13:36:10.380 INFO 10 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2023-11-27 13:36:10.384 INFO 10 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-11-27 13:36:10.401 INFO 10 --- [pool-subtask-executor6,554481c3fd1b426b,4d777ffc0273] c.o.o.s.t.b.cluster.UpdateRsListTask : update rootserver_json success, clusterName=darenOB, obClusterId=10 2023-11-27 13:36:10.403 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.a.service.OcpAlarmServiceImpl : delete Done, affectRow=1, id=25984 2023-11-27 13:36:10.409 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-24 10:49:31.0, nextRun=2023-11-24 10:50:00.0, currentDate=Mon Nov 27 13:36:10 CST 2023 2023-11-27 13:36:10.409 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-11-24 10:49:31.0, nextRunTime=2023-11-24 10:50:00.0} 2023-11-27 13:36:10.428 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:6670996, status:FAILED 2023-11-27 13:36:10.437 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000004, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:10.441 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.a.service.OcpAlarmServiceImpl : create alarm event Done, affectRow=1, id=2000002 2023-11-27 13:36:10.443 INFO 10 --- [pool-Check host status13,,8714d1c389f5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.11:62888/api/v1/agent/status, params:null 2023-11-27 13:36:10.454 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:6670996, status:FAILED 2023-11-27 13:36:10.466 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-11-27 13:36:10.466 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:10.466 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:10.466 INFO 10 --- [pool-Check host status13,,8714d1c389f5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=133.197.204.11 2023-11-27 13:36:10.467 INFO 10 --- [pool-Check host status13,,8714d1c389f5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=133.197.204.11 2023-11-27 13:36:10.467 INFO 10 --- [pool-Check host status13,,8714d1c389f5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=133.197.204.11 2023-11-27 13:36:10.482 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000002, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000002, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:09.676+08:00, sub_task_instance_id=10000002}, listMap={}}, executor=10.120.0.1 2023-11-27 13:36:10.490 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.a.service.OcpAlarmServiceImpl : create alarm event history Done, affectRow=1, id=2000002 2023-11-27 13:36:10.517 INFO 10 --- [pool-Check host status12,,1b34771788d8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.10:62888/api/v1/agent/status, params:null 2023-11-27 13:36:10.533 INFO 10 --- [pool-subtask-executor4,554481c3fd1b426b,a8a0ae9d6cc0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.12:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(monagent.ob.monitor.password=xxx (monagent.host.ip=133.197.204.12), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.ob.cluster.name=darenOB), (monagent.ob.cluster.id=10), (monagent.cluster.id=2), (monagent.ob.zone.name=zone5), (monagent.ob.sql.port=2881), (monagent.ob.rpc.port=2882), (ob.log.path=/home/obdata/log), (ob.data.path=/home/obdata/data), (monagent.pipeline.ob.status=active), (ob.logcleaner.enabled=true), (ob.logtailer.enabled=true), (ob.install.path=/home/obdata/admin/oceanbase), (monagent.log.level=info), (mgragent.log.level=info), (monagent.monitordb.host=133.197.206.97), (monagent.monitordb.port=2883), (monagent.monitordb.user=monitor_user@monitor_tenant#obcluster), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.obproxy.cluster.id=4), (monagent.obproxy.cluster.name=darenOBProxy), (monagent.host.ip=133.197.204.12), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.log.level=info), (mgragent.log.level=info), (monagent.obproxy.sql.port=2883), (monagent.pipeline.obproxy.status=active), (monagent.obproxy.exporter.port=2884), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["is_out_of_memstore_mem=true","right_to_die_or_duty_to_live","on_fatal_error"],"level":1,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["partition table update task cost too much time to execute","clog disk is almost full","set_disk_error:attention!!!","error occurs on data disk or slog disk"],"level":2,"logLevels":["ERROR","WARNING"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"election_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"election_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_all_error_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_expanse_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_expanse_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"obproxy_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"obproxy_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_sql_execute_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"sql_execute_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_compress_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_compress_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"message_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["kernel: nvme"],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"rootservice_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"rootservice_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_sql_query_slow","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"slow_sql_query_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300}]), (monagent.pipeline.sql.audit.status=active), (monagent.pipeline.sql.plan.status=active), (monagent.pipeline.plan.monitor.status=active), (monagent.pipeline.transaction.status=active), (monagent.pipeline.session.status=active), (monagent.pipeline.slow.sql.status=active)]), params:null 2023-11-27 13:36:10.535 INFO 10 --- [pool-Check host status12,,1b34771788d8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=133.197.204.10 2023-11-27 13:36:10.535 INFO 10 --- [pool-Check host status12,,1b34771788d8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=133.197.204.10 2023-11-27 13:36:10.535 INFO 10 --- [pool-Check host status12,,1b34771788d8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=133.197.204.10 2023-11-27 13:36:10.542 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000004, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-11-27T13:36:10.413+08:00, endTime=null} 2023-11-27 13:36:10.552 INFO 10 --- [pool-Check host status11,,d672cf4ce3c1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.92:62888/api/v1/agent/status, params:null 2023-11-27 13:36:10.557 WARN 10 --- [pool-Check host status11,,d672cf4ce3c1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent status check failed. host=3, ip=133.197.206.92, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.92:62888 [/133.197.206.92] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.agentStatus(AgentExecutor.java:622) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostAgentStatus(HostCheckManagerImpl.java:83) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.92:62888 [/133.197.206.92] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 23 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 32 common frames omitted 2023-11-27 13:36:10.563 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-24 10:45:03.0, nextRun=2023-11-24 10:50:00.0, currentDate=Mon Nov 27 13:36:10 CST 2023 2023-11-27 13:36:10.563 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=35, name=Collect all tenants mini compactions, templateName=CollectTenantMiniCompactionSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2023-11-24 10:45:03.0, nextRunTime=2023-11-24 10:50:00.0} 2023-11-27 13:36:10.567 INFO 10 --- [pool-subtask-executor6,554481c3fd1b426b,4d777ffc0273] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000010, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:10.573 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=6, ip=133.197.204.12] desired exporter count: 8, actual exporter count: 0 2023-11-27 13:36:10.573 WARN 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Send exporter not enough alarm, ip=133.197.204.12, deployedServices=[OB_CLUSTER, OB_PROXY], expectedCount=8, activeCount=0 2023-11-27 13:36:10.574 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.s.OcpAlarmInspectService : skipped due no expired filter found 2023-11-27 13:36:10.603 INFO 10 --- [pool-Check host status11,,d672cf4ce3c1] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [3:133.197.206.92], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:36:10.603 INFO 10 --- [pool-Check host status11,,d672cf4ce3c1] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:36:10.646 INFO 10 --- [pool-Check host status11,,d672cf4ce3c1] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=3 2023-11-27 13:36:10.646 INFO 10 --- [pool-Check host status11,,d672cf4ce3c1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2023-11-27 13:36:10.649 INFO 10 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.11:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(monagent.ob.monitor.password=xxx (monagent.host.ip=133.197.204.11), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.ob.cluster.name=darenOB), (monagent.ob.cluster.id=10), (monagent.cluster.id=2), (monagent.ob.zone.name=zone4), (monagent.ob.sql.port=2881), (monagent.ob.rpc.port=2882), (ob.log.path=/home/obdata/log), (ob.data.path=/home/obdata/data), (monagent.pipeline.ob.status=active), (ob.logcleaner.enabled=true), (ob.logtailer.enabled=true), (ob.install.path=/home/obdata/admin/oceanbase), (monagent.log.level=info), (mgragent.log.level=info), (monagent.monitordb.host=133.197.206.97), (monagent.monitordb.port=2883), (monagent.monitordb.user=monitor_user@monitor_tenant#obcluster), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.obproxy.cluster.id=4), (monagent.obproxy.cluster.name=darenOBProxy), (monagent.host.ip=133.197.204.11), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.log.level=info), (mgragent.log.level=info), (monagent.obproxy.sql.port=2883), (monagent.pipeline.obproxy.status=active), (monagent.obproxy.exporter.port=2884), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["is_out_of_memstore_mem=true","right_to_die_or_duty_to_live","on_fatal_error"],"level":1,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["partition table update task cost too much time to execute","clog disk is almost full","set_disk_error:attention!!!","error occurs on data disk or slog disk"],"level":2,"logLevels":["ERROR","WARNING"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"election_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"election_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_all_error_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_expanse_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_expanse_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"obproxy_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"obproxy_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_sql_execute_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"sql_execute_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_compress_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_compress_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"message_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["kernel: nvme"],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"rootservice_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"rootservice_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_sql_query_slow","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"slow_sql_query_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300}]), (monagent.pipeline.sql.audit.status=active), (monagent.pipeline.sql.plan.status=active), (monagent.pipeline.plan.monitor.status=active), (monagent.pipeline.transaction.status=active), (monagent.pipeline.session.status=active), (monagent.pipeline.slow.sql.status=active)]), params:null 2023-11-27 13:36:10.650 INFO 10 --- [pool-subtask-executor4,554481c3fd1b426b,a8a0ae9d6cc0] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 6, do nothing 2023-11-27 13:36:10.655 INFO 10 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.13:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(monagent.ob.monitor.password=xxx (monagent.host.ip=133.197.204.13), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.ob.cluster.name=darenOB), (monagent.ob.cluster.id=10), (monagent.cluster.id=2), (monagent.ob.zone.name=zone6), (monagent.ob.sql.port=2881), (monagent.ob.rpc.port=2882), (ob.log.path=/home/obdata/log), (ob.data.path=/home/obdata/data), (monagent.pipeline.ob.status=active), (ob.logcleaner.enabled=true), (ob.logtailer.enabled=true), (ob.install.path=/home/obdata/admin/oceanbase), (monagent.log.level=info), (mgragent.log.level=info), (monagent.monitordb.host=133.197.206.97), (monagent.monitordb.port=2883), (monagent.monitordb.user=monitor_user@monitor_tenant#obcluster), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.obproxy.cluster.id=4), (monagent.obproxy.cluster.name=darenOBProxy), (monagent.host.ip=133.197.204.13), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.log.level=info), (mgragent.log.level=info), (monagent.obproxy.sql.port=2883), (monagent.pipeline.obproxy.status=active), (monagent.obproxy.exporter.port=2884), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["is_out_of_memstore_mem=true","right_to_die_or_duty_to_live","on_fatal_error"],"level":1,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["partition table update task cost too much time to execute","clog disk is almost full","set_disk_error:attention!!!","error occurs on data disk or slog disk"],"level":2,"logLevels":["ERROR","WARNING"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"election_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"election_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_all_error_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_expanse_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_expanse_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"obproxy_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"obproxy_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_sql_execute_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"sql_execute_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_compress_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_compress_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"message_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["kernel: nvme"],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"rootservice_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"rootservice_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_sql_query_slow","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"slow_sql_query_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300}]), (monagent.pipeline.sql.audit.status=active), (monagent.pipeline.sql.plan.status=active), (monagent.pipeline.plan.monitor.status=active), (monagent.pipeline.transaction.status=active), (monagent.pipeline.session.status=active), (monagent.pipeline.slow.sql.status=active)]), params:null 2023-11-27 13:36:10.657 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants mini compactions, nodeCount=1, taskCount=1 2023-11-27 13:36:10.657 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:10.657 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants mini compactions, taskDefinitionId=35, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:10.659 INFO 10 --- [pool-Check host status11,,d672cf4ce3c1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:36:10.688 INFO 10 --- [pool-Check host status9,,c0d7583006ad] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.97:62888/api/v1/agent/status, params:null 2023-11-27 13:36:10.702 INFO 10 --- [pool-Check host status14,,ae847bb07967] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.12:62888/api/v1/agent/status, params:null 2023-11-27 13:36:10.702 WARN 10 --- [pool-Check host status9,,c0d7583006ad] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent status check failed. host=1, ip=133.197.206.97, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.97:62888 [/133.197.206.97] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.agentStatus(AgentExecutor.java:622) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostAgentStatus(HostCheckManagerImpl.java:83) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.97:62888 [/133.197.206.97] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 23 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 32 common frames omitted 2023-11-27 13:36:10.713 INFO 10 --- [pool-Check host status9,,c0d7583006ad] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [1:133.197.206.97], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:36:10.713 INFO 10 --- [pool-Check host status9,,c0d7583006ad] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:36:10.716 INFO 10 --- [pool-Check host status14,,ae847bb07967] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=133.197.204.12 2023-11-27 13:36:10.716 INFO 10 --- [pool-Check host status14,,ae847bb07967] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=133.197.204.12 2023-11-27 13:36:10.716 INFO 10 --- [pool-Check host status14,,ae847bb07967] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=133.197.204.12 2023-11-27 13:36:10.720 INFO 10 --- [pool-Check host status9,,c0d7583006ad] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=1 2023-11-27 13:36:10.720 INFO 10 --- [pool-Check host status9,,c0d7583006ad] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2023-11-27 13:36:10.721 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.a.service.OcpAlarmServiceImpl : delete Done, affectRow=1, id=31269 2023-11-27 13:36:10.724 INFO 10 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 5, do nothing 2023-11-27 13:36:10.724 INFO 10 --- [pool-Check host status9,,c0d7583006ad] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:36:10.726 INFO 10 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 7, do nothing 2023-11-27 13:36:10.729 INFO 10 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.10:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(monagent.ob.monitor.password=xxx (monagent.host.ip=133.197.204.10), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.ob.cluster.name=darenOB), (monagent.ob.cluster.id=10), (monagent.cluster.id=2), (monagent.ob.zone.name=zone3), (monagent.ob.sql.port=2881), (monagent.ob.rpc.port=2882), (ob.log.path=/home/obdata/log), (ob.data.path=/home/obdata/data), (monagent.pipeline.ob.status=active), (ob.logcleaner.enabled=true), (ob.logtailer.enabled=true), (ob.install.path=/home/obdata/admin/oceanbase), (monagent.log.level=info), (mgragent.log.level=info), (monagent.monitordb.host=133.197.206.97), (monagent.monitordb.port=2883), (monagent.monitordb.user=monitor_user@monitor_tenant#obcluster), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.obproxy.cluster.id=4), (monagent.obproxy.cluster.name=darenOBProxy), (monagent.host.ip=133.197.204.10), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.log.level=info), (mgragent.log.level=info), (monagent.obproxy.sql.port=2883), (monagent.pipeline.obproxy.status=active), (monagent.obproxy.exporter.port=2884), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["is_out_of_memstore_mem=true","right_to_die_or_duty_to_live","on_fatal_error"],"level":1,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["partition table update task cost too much time to execute","clog disk is almost full","set_disk_error:attention!!!","error occurs on data disk or slog disk"],"level":2,"logLevels":["ERROR","WARNING"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"election_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"election_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_all_error_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_expanse_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_expanse_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"obproxy_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"obproxy_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_sql_execute_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"sql_execute_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_compress_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_compress_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"message_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["kernel: nvme"],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"rootservice_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"rootservice_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_sql_query_slow","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"slow_sql_query_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300}]), (monagent.pipeline.sql.audit.status=active), (monagent.pipeline.sql.plan.status=active), (monagent.pipeline.plan.monitor.status=active), (monagent.pipeline.transaction.status=active), (monagent.pipeline.session.status=active), (monagent.pipeline.slow.sql.status=active)]), params:null 2023-11-27 13:36:10.743 INFO 10 --- [pool-subtask-executor10,554481c3fd1b426b,f0527ccaa7bb] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000003, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000003, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:09.975+08:00, sub_task_instance_id=10000003}, listMap={}}, executor=10.120.0.1 2023-11-27 13:36:10.748 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-24 01:00:13.0, nextRun=2023-11-25 01:00:00.0, currentDate=Mon Nov 27 13:36:10 CST 2023 2023-11-27 13:36:10.748 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=36, name=Clean table records, templateName=TableCleaningSchedule, scheduleRule=0 0 1 * * *, lastRunTime=2023-11-24 01:00:13.0, nextRunTime=2023-11-25 01:00:00.0} 2023-11-27 13:36:10.749 INFO 10 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 4, do nothing 2023-11-27 13:36:10.754 INFO 10 --- [pool-Check host status15,,da4a0c4b6da0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.13:62888/api/v1/agent/status, params:null 2023-11-27 13:36:10.755 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.a.service.OcpAlarmServiceImpl : create alarm event Done, affectRow=1, id=2000003 2023-11-27 13:36:10.769 INFO 10 --- [pool-Check host status15,,da4a0c4b6da0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=133.197.204.13 2023-11-27 13:36:10.769 INFO 10 --- [pool-Check host status15,,da4a0c4b6da0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=133.197.204.13 2023-11-27 13:36:10.769 INFO 10 --- [pool-Check host status15,,da4a0c4b6da0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=133.197.204.13 2023-11-27 13:36:10.791 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Clean table records, nodeCount=1, taskCount=1 2023-11-27 13:36:10.791 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:10.791 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Clean table records, taskDefinitionId=36, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:10.806 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.a.service.OcpAlarmServiceImpl : create alarm event history Done, affectRow=1, id=2000003 2023-11-27 13:36:10.852 INFO 10 --- [pool-Check host status16,,caf7ad2317c3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.88:62888/api/v1/agent/status, params:null 2023-11-27 13:36:10.861 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=7, ip=133.197.204.13] desired exporter count: 8, actual exporter count: 0 2023-11-27 13:36:10.861 WARN 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Send exporter not enough alarm, ip=133.197.204.13, deployedServices=[OB_CLUSTER, OB_PROXY], expectedCount=8, activeCount=0 2023-11-27 13:36:10.858 WARN 10 --- [pool-Check host status16,,caf7ad2317c3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent status check failed. host=9, ip=133.197.206.88, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.88:62888 [/133.197.206.88] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.agentStatus(AgentExecutor.java:622) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostAgentStatus(HostCheckManagerImpl.java:83) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.88:62888 [/133.197.206.88] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 23 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 32 common frames omitted 2023-11-27 13:36:10.875 INFO 10 --- [pool-Check host status10,,4382b94d5e22] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.3:62888/api/v1/agent/status, params:null 2023-11-27 13:36:10.877 INFO 10 --- [pool-Check host status16,,caf7ad2317c3] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [9:133.197.206.88], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:36:10.878 INFO 10 --- [pool-Check host status16,,caf7ad2317c3] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:36:10.885 WARN 10 --- [pool-Check host status10,,4382b94d5e22] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent status check failed. host=2, ip=133.197.204.3, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.204.3:62888 [/133.197.204.3] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.agentStatus(AgentExecutor.java:622) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostAgentStatus(HostCheckManagerImpl.java:83) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.204.3:62888 [/133.197.204.3] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 23 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 32 common frames omitted 2023-11-27 13:36:10.892 INFO 10 --- [pool-subtask-executor4,554481c3fd1b426b,a8a0ae9d6cc0] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000009, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:10.893 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-24 10:45:03.0, nextRun=2023-11-24 10:50:00.0, currentDate=Mon Nov 27 13:36:10 CST 2023 2023-11-27 13:36:10.893 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=37, name=HighRiskSqlSchedule, templateName=HighRiskSqlSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2023-11-24 10:45:03.0, nextRunTime=2023-11-24 10:50:00.0} 2023-11-27 13:36:10.924 INFO 10 --- [pool-Check host status10,,4382b94d5e22] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [2:133.197.204.3], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:36:10.924 INFO 10 --- [pool-Check host status10,,4382b94d5e22] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:36:10.926 INFO 10 --- [pool-Check host status16,,caf7ad2317c3] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=9 2023-11-27 13:36:10.926 INFO 10 --- [pool-Check host status16,,caf7ad2317c3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=9 2023-11-27 13:36:10.954 WARN 10 --- [pool-subtask-executor8,554481c3fd1b426b,7a5a3ce68651] c.o.o.c.d.lock.jdbc.DefaultLockManager : LockException while executing with jdbc lock part-worker Cannot acquire lock for key part-worker 2023-11-27 13:36:10.955 WARN 10 --- [pool-subtask-executor8,554481c3fd1b426b,7a5a3ce68651] c.o.o.c.t.engine.runner.RunnerFactory : Execute task failed, subtask=SubtaskInstanceOverview{id=10000001, name=Partition rollover daemon, state=FAILED, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=2023-11-27T13:36:09.490+08:00, endTime=null} com.oceanbase.ocp.core.distributed.lock.LockException: Cannot acquire lock for key part-worker at com.oceanbase.ocp.core.distributed.lock.jdbc.DefaultLockManager.doExecuteWhileLocked(DefaultLockManager.java:97) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.distributed.lock.jdbc.DefaultLockManager.executeWhileLocked(DefaultLockManager.java:86) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.service.partitioning.PartitionRolloverService.watchAllWhileLocked(PartitionRolloverService.java:122) ~[ocp-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task.run(PartitioningDaemonSchedule.java:42) ~[ocp-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:60) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:111) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.redirectOutputIfNotSysSchedule(RunnerFactory.java:174) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.run(RunnerFactory.java:101) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.ReadySubtaskWorker.lambda$null$1(ReadySubtaskWorker.java:127) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] 2023-11-27 13:36:10.963 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000005, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:10.974 INFO 10 --- [pool-Check host status16,,caf7ad2317c3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:36:10.976 INFO 10 --- [pool-Check host status10,,4382b94d5e22] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=2 2023-11-27 13:36:10.976 INFO 10 --- [pool-Check host status10,,4382b94d5e22] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2023-11-27 13:36:11.002 INFO 10 --- [pool-subtask-executor8,554481c3fd1b426b,7a5a3ce68651] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000001, operation:EXECUTE, state: FAILED 2023-11-27 13:36:11.015 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.a.service.OcpAlarmServiceImpl : delete Done, affectRow=1, id=25986 2023-11-27 13:36:11.015 INFO 10 --- [pool-Check host status10,,4382b94d5e22] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:36:11.037 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000005, name=Wait node, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=0, startTime=2023-11-27T13:36:10.896+08:00, endTime=null} 2023-11-27 13:36:11.050 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.a.service.OcpAlarmServiceImpl : create alarm event Done, affectRow=1, id=2000004 2023-11-27 13:36:11.061 INFO 10 --- [pool-subtask-executor11,554481c3fd1b426b,19d534b19150] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000004, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000004, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:10.402+08:00, sub_task_instance_id=10000004}, listMap={}}, executor=10.120.0.1 2023-11-27 13:36:11.066 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.a.service.OcpAlarmServiceImpl : create alarm event history Done, affectRow=1, id=2000004 2023-11-27 13:36:11.074 INFO 10 --- [pool-subtask-executor11,554481c3fd1b426b,19d534b19150] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-11-27 13:36:11.075 INFO 10 --- [ForkJoinPool-12-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-11-27 13:36:11.106 WARN 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.s.t.b.host.CheckExporterTask : Host [id=9, ip=133.197.206.88, status=OFFLINE] not online, skip check exporter 2023-11-27 13:36:11.128 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:11.128 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:11.136 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2023-11-27 13:36:11.141 INFO 10 --- [pool-subtask-executor11,554481c3fd1b426b,19d534b19150] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000004, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:11.172 INFO 10 --- [pool-Check obproxy1,554481c3fd1b426b,9f4e6f2d835c] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-11-27 13:36:11.186 WARN 10 --- [pool-Check obproxy1,554481c3fd1b426b,9f4e6f2d835c] c.oceanbase.ocp.odpsdk.OdpJdbcOperator : Failed to check odp connection Connection {driverClassName ='com.oceanbase.jdbc.Driver', connectionProperties ='[user=root@proxysys, url=jdbc:oceanbase://133.197.204.3:2883/?connectProxy=true]'}, error is Failed to obtain JDBC Connection; nested exception is java.sql.SQLNonTransientConnectionException: Could not connect to HostAddress{host='133.197.204.3', port=2883}. Connection refused (Connection refused) 2023-11-27 13:36:11.186 WARN 10 --- [pool-Check obproxy1,554481c3fd1b426b,9f4e6f2d835c] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to connect obproxy server, maybe caused by odp process down, wrong password, or ocp address not in private ip list and odp parameter skip_proxy_sys_private_check is not true 2023-11-27 13:36:11.187 ERROR 10 --- [pool-Check obproxy1,554481c3fd1b426b,9f4e6f2d835c] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.UnexpectedException occurred with code error.obproxy.connection.error, and args [133.197.204.3:2883] 2023-11-27 13:36:11.201 INFO 10 --- [pool-subtask-executor2,554481c3fd1b426b,45d15d0ce112] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000008, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:11.219 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2023-11-27 13:36:11.226 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=HighRiskSqlAuditStatTaskSchedule, nodeCount=1, taskCount=1 2023-11-27 13:36:11.226 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:11.226 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=HighRiskSqlAuditStatTaskSchedule, taskDefinitionId=37, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:11.229 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20231025 2023-11-27 13:36:11.233 INFO 10 --- [pool-Check obproxy3,554481c3fd1b426b,c3b65c9c2dca] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-11-27 13:36:11.235 WARN 10 --- [pool-Check obproxy1,554481c3fd1b426b,9f4e6f2d835c] c.oceanbase.ocp.odpsdk.OdpJdbcOperator : Failed to check odp connection Connection {driverClassName ='com.oceanbase.jdbc.Driver', connectionProperties ='[user=root@proxysys, url=jdbc:oceanbase://133.197.204.3:2883/?connectProxy=true]'}, error is Failed to obtain JDBC Connection; nested exception is java.sql.SQLNonTransientConnectionException: Could not connect to HostAddress{host='133.197.204.3', port=2883}. Connection refused (Connection refused) 2023-11-27 13:36:11.235 WARN 10 --- [pool-Check obproxy1,554481c3fd1b426b,9f4e6f2d835c] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to connect obproxy server, maybe caused by odp process down, wrong password, or ocp address not in private ip list and odp parameter skip_proxy_sys_private_check is not true 2023-11-27 13:36:11.235 ERROR 10 --- [pool-Check obproxy1,554481c3fd1b426b,9f4e6f2d835c] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.UnexpectedException occurred with code error.obproxy.connection.error, and args [133.197.204.3:2883] 2023-11-27 13:36:11.243 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:11.243 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:11.245 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` drop partition (P20231025) 2023-11-27 13:36:11.248 INFO 10 --- [pool-Check obproxy2,554481c3fd1b426b,6af3a2988277] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-11-27 13:36:11.270 INFO 10 --- [pool-Check obproxy5,554481c3fd1b426b,544afd1e6a25] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-11-27 13:36:11.287 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-24 01:00:14.0, nextRun=2023-11-25 01:00:00.0, currentDate=Mon Nov 27 13:36:11 CST 2023 2023-11-27 13:36:11.288 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=1000001, name=Clean table records, templateName=TableCleaningSchedule, scheduleRule=0 0 1 * * *, lastRunTime=2023-11-24 01:00:14.0, nextRunTime=2023-11-25 01:00:00.0} 2023-11-27 13:36:11.288 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000007, status:SUCCESS 2023-11-27 13:36:11.289 INFO 10 --- [pool-Check obproxy4,554481c3fd1b426b,30f15b6b3345] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-11-27 13:36:11.297 INFO 10 --- [pool-Check obproxy3,554481c3fd1b426b,c3b65c9c2dca] c.o.o.s.o.obproxy.ObproxyService : check obproxy 10 success, update last active time to now 2023-11-27 13:36:11.315 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000007, status:SUCCESS 2023-11-27 13:36:11.329 INFO 10 --- [pool-Check obproxy2,554481c3fd1b426b,6af3a2988277] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-11-27 13:36:11.335 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Clean table records, nodeCount=1, taskCount=1 2023-11-27 13:36:11.335 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:11.335 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Clean table records, taskDefinitionId=1000001, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:11.380 INFO 10 --- [pool-Check obproxy5,554481c3fd1b426b,544afd1e6a25] c.o.o.s.o.obproxy.ObproxyService : check obproxy 12 success, update last active time to now 2023-11-27 13:36:11.380 INFO 10 --- [pool-Check obproxy4,554481c3fd1b426b,30f15b6b3345] c.o.o.s.o.obproxy.ObproxyService : check obproxy 11 success, update last active time to now 2023-11-27 13:36:11.384 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-24 10:49:01.0, nextRun=2023-11-24 10:50:00.0, currentDate=Mon Nov 27 13:36:11 CST 2023 2023-11-27 13:36:11.384 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=1000002, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-24 10:49:01.0, nextRunTime=2023-11-24 10:50:00.0} 2023-11-27 13:36:11.397 INFO 10 --- [pool-Check obproxy1,554481c3fd1b426b,9f4e6f2d835c] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 failed, will not update last active time com.oceanbase.ocp.core.exception.UnexpectedException: [OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=OBPROXY_CONNECTION_ERROR, args=133.197.204.3:2883 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_312] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_312] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_312] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_312] at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.util.ExceptionUtils.throwException(ExceptionUtils.java:90) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.util.ExceptionUtils.unExpected(ExceptionUtils.java:77) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.service.operation.obproxy.SystemOdpOperatorFactory.createOdpOperator(SystemOdpOperatorFactory.java:105) ~[ocp-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.service.operation.obproxy.SystemOdpOperatorFactory.tryToCreateOdpOperator(SystemOdpOperatorFactory.java:86) ~[ocp-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.service.operation.obproxy.ObproxyService.lambda$checkObproxyServers$20(ObproxyService.java:517) ~[ocp-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] 2023-11-27 13:36:11.418 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-11-27 13:36:11.418 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:11.418 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=1000002, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:11.437 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000008, status:SUCCESS 2023-11-27 13:36:11.447 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000008, status:SUCCESS 2023-11-27 13:36:11.470 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-24 00:00:05.0, nextRun=2023-11-25 00:00:00.0, currentDate=Mon Nov 27 13:36:11 CST 2023 2023-11-27 13:36:11.470 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=1000003, name=Clean expired log files on schedule, templateName=BackupScheduleCleanLogTemplateFactory, scheduleRule=0 0 0 * * *, lastRunTime=2023-11-24 00:00:05.0, nextRunTime=2023-11-25 00:00:00.0} 2023-11-27 13:36:11.484 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Clean expired log files, nodeCount=1, taskCount=1 2023-11-27 13:36:11.485 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:11.485 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Clean expired log files, taskDefinitionId=1000003, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:11.506 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:11.507 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:11.519 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2023-11-27 13:36:11.534 INFO 10 --- [pool-subtask-executor10,554481c3fd1b426b,f0527ccaa7bb] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000003, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:11.539 INFO 10 --- [ForkJoinPool-12-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:11.545 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20231026 2023-11-27 13:36:11.550 INFO 10 --- [ForkJoinPool-12-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:11.550 INFO 10 --- [ForkJoinPool-12-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:11.551 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-24 09:00:03.0, nextRun=2023-11-24 12:00:00.0, currentDate=Mon Nov 27 13:36:11 CST 2023 2023-11-27 13:36:11.551 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=1000004, name=Collect backup files size on schedule, templateName=BackupScheduleCollectTemplateFactory, scheduleRule=0 0 */3 * * *, lastRunTime=2023-11-24 09:00:03.0, nextRunTime=2023-11-24 12:00:00.0} 2023-11-27 13:36:11.551 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:11.551 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:11.554 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` drop partition (P20231026) 2023-11-27 13:36:11.560 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect backup files size, nodeCount=1, taskCount=1 2023-11-27 13:36:11.560 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:11.560 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect backup files size, taskDefinitionId=1000004, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:11.560 INFO 10 --- [ForkJoinPool-12-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:11.581 INFO 10 --- [ForkJoinPool-12-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:11.581 INFO 10 --- [ForkJoinPool-12-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:11.592 INFO 10 --- [ForkJoinPool-12-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT MAX(CASE `name` WHEN 'cluster' THEN `info` ELSE '' END ) `cluster`, MAX(CASE `name` WHEN 'merge_status' THEN `info` ELSE '' END ) `merge_status`, MAX(CASE `name` WHEN 'global_broadcast_version' THEN `value` ELSE 0 END ) `global_broadcast_version`, MAX(CASE `name` WHEN 'frozen_version' THEN `value` ELSE 0 END ) `frozen_version`, MAX(CASE `name` WHEN 'last_merged_version' THEN `value` ELSE 0 END ) `last_merged_version`, MAX(CASE `name` WHEN 'frozen_time' THEN `value` ELSE 0 END ) `frozen_time`, MAX(CASE `name` WHEN 'is_merge_error' THEN `value` ELSE 0 END ) `merge_error` FROM oceanbase.__all_zone WHERE `zone` = '' 2023-11-27 13:36:11.610 INFO 10 --- [ForkJoinPool-12-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:11.610 INFO 10 --- [ForkJoinPool-12-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:11.619 INFO 10 --- [ForkJoinPool-12-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select column_value from oceanbase.__all_core_table where table_name ='__all_global_stat' and column_name = 'frozen_version' 2023-11-27 13:36:11.643 INFO 10 --- [ForkJoinPool-12-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:11.644 INFO 10 --- [ForkJoinPool-12-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:11.647 INFO 10 --- [ForkJoinPool-12-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, MAX(CASE `name` WHEN 'region' THEN `info` ELSE '' END ) `region`, MAX(CASE `name` WHEN 'idc' THEN `info` ELSE '' END ) `idc`, MAX(CASE `name` WHEN 'status' THEN `info` ELSE '' END ) `status`, MAX(CASE `name` WHEN 'merge_status' THEN `info` ELSE '' END ) `merge_status`, MAX(CASE `name` WHEN 'broadcast_version' THEN `value` ELSE 0 END ) `broadcast_version`, MAX(CASE `name` WHEN 'all_merged_version' THEN `value` ELSE 0 END ) `all_merged_version`, MAX(CASE `name` WHEN 'last_merged_version' THEN `value` ELSE 0 END ) `last_merged_version`, MAX(CASE `name` WHEN 'merge_start_time' THEN `value` ELSE 0 END ) `merge_start_time`, MAX(CASE `name` WHEN 'last_merged_time' THEN `value` ELSE 0 END ) `last_merged_time`, MAX(CASE `name` WHEN 'is_merge_timeout' THEN `value` ELSE 0 END ) `merge_timeout` FROM oceanbase.__all_zone WHERE `zone` <> '' GROUP BY `zone` 2023-11-27 13:36:11.665 INFO 10 --- [ForkJoinPool-12-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : compaction collect success, cluster=ObClusterOverview(id=1, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"133.197.206.97:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1700793672036000}, createTime=2022-11-29T14:52:53+08:00, creator=admin, obVersion=3.1.4, tenantCount=3) 2023-11-27 13:36:11.695 INFO 10 --- [pool-subtask-executor12,554481c3fd1b426b,8778043a9a55] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000005, context=Context{parallelIdx=-1, stringMap={begin_interval_time=2023-11-27T12:36:06.973+08:00, prohibit_rollback=false, task_instance_id=8000005, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:10.880+08:00, end_interval_time=2023-11-27T13:36:06.973+08:00, sub_task_instance_id=10000005}, listMap={cluster_ids=[1, 2]}}, executor=10.120.0.1 2023-11-27 13:36:11.714 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:11.715 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:11.723 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2023-11-27 13:36:11.771 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20231027 2023-11-27 13:36:11.783 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:11.783 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:11.790 INFO 10 --- [pool-subtask-executor12,554481c3fd1b426b,8778043a9a55] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000005, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:11.793 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` drop partition (P20231027) 2023-11-27 13:36:11.813 INFO 10 --- [pool-overview-container-8,,1a3fbfb53615] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=4 2023-11-27 13:36:11.814 INFO 10 --- [pool-overview-container-8,,1a3fbfb53615] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=16781 2023-11-27 13:36:11.900 INFO 10 --- [pool-scanner--1135-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=132 2023-11-27 13:36:11.960 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:11.960 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:11.965 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2023-11-27 13:36:11.989 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:11.989 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:11.996 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.s.OcpAlarmInspectService : Archive expired alarm history data: archive event success, affectRows=896 2023-11-27 13:36:11.996 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.s.OcpAlarmInspectService : delete expired alarm events count 896 2023-11-27 13:36:12.009 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:12.038 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:12.039 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:12.073 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2023-11-27 13:36:12.126 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20231203 2023-11-27 13:36:12.137 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:12.137 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:12.147 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20231203 values less than (1701648000000000)) 2023-11-27 13:36:12.282 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000024, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:12.292 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000024, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-11-27T13:36:12.277+08:00, endTime=null} 2023-11-27 13:36:12.355 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000025, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:12.423 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000025, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask, seriesId=0, startTime=2023-11-27T13:36:12.348+08:00, endTime=null} 2023-11-27 13:36:12.425 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:12.426 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:12.429 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2023-11-27 13:36:12.460 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2023-11-27 13:36:12.473 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20231204 2023-11-27 13:36:12.484 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:12.484 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:12.489 INFO 10 --- [pool-subtask-executor13,554481c3fd1b426b,d13906d23d11] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000024, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000018, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:12.277+08:00, sub_task_instance_id=10000024}, listMap={}}, executor=10.120.0.1 2023-11-27 13:36:12.489 INFO 10 --- [pool-subtask-executor13,554481c3fd1b426b,d13906d23d11] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-11-27 13:36:12.492 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20231204 values less than (1701734400000000)) 2023-11-27 13:36:12.580 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000026, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:12.621 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000026, name=Clean table records, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.dataclean.TableCleaningTask, seriesId=0, startTime=2023-11-27T13:36:12.563+08:00, endTime=null} 2023-11-27 13:36:12.626 INFO 10 --- [pool-98-thread-1,554481c3fd1b426b,8a8b19a470f9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:darenOB, tenant:sys 2023-11-27 13:36:12.646 INFO 10 --- [pool-98-thread-2,554481c3fd1b426b,f58a3b1f3bcb] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:darenOB, tenant:daren 2023-11-27 13:36:12.658 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.s.OcpAlarmInspectService : skipped due no expired event found 2023-11-27 13:36:12.658 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.s.OcpAlarmInspectService : delete expired alarm events count 0 2023-11-27 13:36:12.658 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.s.OcpAlarmInspectService : archiveExpiredEvents count 896 2023-11-27 13:36:12.672 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.s.OcpAlarmInspectService : skipped due no expired notification found 2023-11-27 13:36:12.672 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.s.OcpAlarmInspectService : archive notifications count 0 2023-11-27 13:36:12.672 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.s.OcpAlarmInspectService : archiveExpiredNotifications count 0 2023-11-27 13:36:12.770 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:36:12.775 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:36:12.776 INFO 10 --- [pool-subtask-executor14,554481c3fd1b426b,c5ea75c89d96] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000025, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000019, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:12.348+08:00, sub_task_instance_id=10000025}, listMap={}}, executor=10.120.0.1 2023-11-27 13:36:12.776 INFO 10 --- [pool-subtask-executor14,554481c3fd1b426b,c5ea75c89d96] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2023-11-27 13:36:12.809 INFO 10 --- [pool-task-coordinator2,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Mark failed task, entity=TaskInstanceEntity{id=8000001, name=Partition rollover daemon, taskDefinitionId=1, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:12.813 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000027, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:12.815 INFO 10 --- [pool-98-thread-1,554481c3fd1b426b,8a8b19a470f9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:12.824 INFO 10 --- [pool-98-thread-1,554481c3fd1b426b,8a8b19a470f9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:12.824 INFO 10 --- [pool-98-thread-1,554481c3fd1b426b,8a8b19a470f9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:12.832 INFO 10 --- [pool-98-thread-2,554481c3fd1b426b,f58a3b1f3bcb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:12.834 INFO 10 --- [pool-98-thread-2,554481c3fd1b426b,f58a3b1f3bcb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:12.834 INFO 10 --- [pool-98-thread-2,554481c3fd1b426b,f58a3b1f3bcb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:12.851 INFO 10 --- [pool-98-thread-2,554481c3fd1b426b,f58a3b1f3bcb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:12.852 INFO 10 --- [pool-98-thread-1,554481c3fd1b426b,8a8b19a470f9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:12.853 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000027, name=High Risk Sql Audit Stat Task, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.HighRiskSqlAuditStatTask, seriesId=0, startTime=2023-11-27T13:36:12.806+08:00, endTime=null} 2023-11-27 13:36:12.854 INFO 10 --- [pool-ocp-schedules-3,,67e02e0d2018] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:36:12.865 INFO 10 --- [pool-98-thread-2,554481c3fd1b426b,f58a3b1f3bcb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:12.865 INFO 10 --- [pool-98-thread-2,554481c3fd1b426b,f58a3b1f3bcb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:12.874 INFO 10 --- [pool-98-thread-2,554481c3fd1b426b,f58a3b1f3bcb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-11-27 13:36:12.876 INFO 10 --- [pool-98-thread-1,554481c3fd1b426b,8a8b19a470f9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:12.876 INFO 10 --- [pool-98-thread-1,554481c3fd1b426b,8a8b19a470f9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:12.884 INFO 10 --- [pool-98-thread-1,554481c3fd1b426b,8a8b19a470f9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-11-27 13:36:12.930 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000001, status:FAILED 2023-11-27 13:36:12.951 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000001, status:FAILED 2023-11-27 13:36:12.976 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,505d93a36926] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000026, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000020, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:12.553+08:00, sub_task_instance_id=10000026}, listMap={}}, executor=10.120.0.1 2023-11-27 13:36:12.989 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:36:12.981992+08:00, lastProcessTime=2023-11-27T13:36:03.483800+08:00 2023-11-27 13:36:12.993 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000028, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:13.028 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:13.028 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:13.031 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2023-11-27 13:36:13.052 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000028, name=Clean table records, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.dataclean.TableCleaningTask, seriesId=0, startTime=2023-11-27T13:36:12.988+08:00, endTime=null} 2023-11-27 13:36:13.075 INFO 10 --- [pool-98-thread-1,554481c3fd1b426b,8a8b19a470f9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1701021601748952808, broadcast scn:1701021601748952808, saved frozen scn:1700762403842899601 2023-11-27 13:36:13.075 INFO 10 --- [pool-98-thread-1,554481c3fd1b426b,8a8b19a470f9] o.i.t.TenantCompactionCollectServiceImpl : attach new compaction, ready to create compaction info 2023-11-27 13:36:13.090 INFO 10 --- [pool-98-thread-2,554481c3fd1b426b,f58a3b1f3bcb] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1701021600396785720, broadcast scn:1701021600396785720, saved frozen scn:1700762400933391038 2023-11-27 13:36:13.090 INFO 10 --- [pool-98-thread-2,554481c3fd1b426b,f58a3b1f3bcb] o.i.t.TenantCompactionCollectServiceImpl : attach new compaction, ready to create compaction info 2023-11-27 13:36:13.173 INFO 10 --- [pool-98-thread-1,554481c3fd1b426b,8a8b19a470f9] o.i.t.TenantCompactionCollectServiceImpl : create compaction info success, compaction info:ObTenantCompactionEntity(id=2000001, createTime=null, updateTime=null, tenantId=4, status=SUCCESS, startTime=2023-11-27T02:00:04.482686+08:00, lastFinishTime=2023-11-27T02:01:26.045028+08:00, frozenScn=1701021601748952808, statsCollected=false, maxMiniFreezeCount=0, maxMemPercent=0) 2023-11-27 13:36:13.173 INFO 10 --- [pool-98-thread-1,554481c3fd1b426b,8a8b19a470f9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction stats, tenant :sys 2023-11-27 13:36:13.257 INFO 10 --- [pool-subtask-executor16,554481c3fd1b426b,761a8697ecb7] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000027, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=8000021, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:12.798+08:00, sub_task_instance_id=10000027}, listMap={cluster_ids=[[1,1,2]], tenant_ids=[[2,3,5]]}}, executor=10.120.0.1 2023-11-27 13:36:13.260 INFO 10 --- [pool-subtask-executor16,554481c3fd1b426b,761a8697ecb7] c.o.o.p.s.i.t.HighRiskSqlAuditStatTask : start to diagnose[2, 3, 5] 2023-11-27 13:36:13.277 INFO 10 --- [pool-98-thread-2,554481c3fd1b426b,f58a3b1f3bcb] o.i.t.TenantCompactionCollectServiceImpl : create compaction info success, compaction info:ObTenantCompactionEntity(id=2000002, createTime=null, updateTime=null, tenantId=5, status=SUCCESS, startTime=2023-11-27T02:00:00.771087+08:00, lastFinishTime=2023-11-27T02:03:26.250880+08:00, frozenScn=1701021600396785720, statsCollected=false, maxMiniFreezeCount=0, maxMemPercent=0) 2023-11-27 13:36:13.277 INFO 10 --- [pool-98-thread-2,554481c3fd1b426b,f58a3b1f3bcb] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction stats, tenant :daren 2023-11-27 13:36:13.327 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000003, status:SUCCESS 2023-11-27 13:36:13.331 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000003, status:SUCCESS 2023-11-27 13:36:13.342 INFO 10 --- [pool-101-thread-1,554481c3fd1b426b,cba9b1354064] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2023-11-27 13:36:13.361 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2023-11-27 13:36:13.365 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000029, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:13.378 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20231205 2023-11-27 13:36:13.391 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:13.391 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:13.415 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20231205 values less than (1701820800000000)) 2023-11-27 13:36:13.424 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000029, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-11-27T13:36:13.328+08:00, endTime=null} 2023-11-27 13:36:13.442 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000004, status:SUCCESS 2023-11-27 13:36:13.444 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000004, status:SUCCESS 2023-11-27 13:36:13.459 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:36:13.495 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:36:13.617 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000006, status:SUCCESS 2023-11-27 13:36:13.617 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,07ed6be6b585] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000028, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000022, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:12.988+08:00, sub_task_instance_id=10000028}, listMap={}}, executor=10.120.0.1 2023-11-27 13:36:13.632 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000006, status:SUCCESS 2023-11-27 13:36:13.685 INFO 10 --- [pool-scanner--1137-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=300 2023-11-27 13:36:13.701 INFO 10 --- [pool-scanner--1136-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=294 2023-11-27 13:36:13.727 INFO 10 --- [pool-101-thread-1,554481c3fd1b426b,cba9b1354064] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:13.728 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000030, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:13.732 INFO 10 --- [pool-101-thread-1,554481c3fd1b426b,cba9b1354064] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:13.732 INFO 10 --- [pool-101-thread-1,554481c3fd1b426b,cba9b1354064] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:13.733 INFO 10 --- [pool-101-thread-1,554481c3fd1b426b,cba9b1354064] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:13.825 INFO 10 --- [pool-scanner--1140-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=294 2023-11-27 13:36:13.833 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:36:13.862 INFO 10 --- [pool-scanner--1141-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=294 2023-11-27 13:36:13.865 INFO 10 --- [pool-98-thread-1,554481c3fd1b426b,8a8b19a470f9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction stats, tenantId:sys 2023-11-27 13:36:13.865 INFO 10 --- [pool-98-thread-1,554481c3fd1b426b,8a8b19a470f9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:darenOB, tenant:sys 2023-11-27 13:36:13.897 INFO 10 --- [pool-scanner--1139-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=300 2023-11-27 13:36:13.931 INFO 10 --- [pool-scanner--1138-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=300 2023-11-27 13:36:13.939 INFO 10 --- [pool-98-thread-2,554481c3fd1b426b,f58a3b1f3bcb] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction stats, tenantId:daren 2023-11-27 13:36:13.939 INFO 10 --- [pool-98-thread-2,554481c3fd1b426b,f58a3b1f3bcb] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:darenOB, tenant:daren 2023-11-27 13:36:13.950 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:13.950 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:13.953 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2023-11-27 13:36:14.019 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000030, name=Clean expired log files, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.CleanLogTask, seriesId=0, startTime=2023-11-27T13:36:13.621+08:00, endTime=null} 2023-11-27 13:36:14.235 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:36:14.254 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:36:14.550 INFO 10 --- [pool-subtask-executor13,554481c3fd1b426b,d13906d23d11] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-11-27 13:36:14.553 INFO 10 --- [pool-ocp-schedules-6,,7d7f21449217] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:36:14.553 INFO 10 --- [pool-ocp-schedules-6,,7d7f21449217] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:36:14.607 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2023-11-27 13:36:14.732 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20231116 2023-11-27 13:36:14.736 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:14.736 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:14.737 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` drop partition (P20231116) 2023-11-27 13:36:14.784 INFO 10 --- [pool-101-thread-1,554481c3fd1b426b,cba9b1354064] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:14.784 INFO 10 --- [pool-101-thread-1,554481c3fd1b426b,cba9b1354064] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:14.790 INFO 10 --- [pool-101-thread-1,554481c3fd1b426b,cba9b1354064] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1700976973000000, 1701063373000000] 2023-11-27 13:36:14.793 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:36:14.794 INFO 10 --- [pool-subtask-executor13,554481c3fd1b426b,d13906d23d11] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000024, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:14.922 INFO 10 --- [pool-subtask-executor18,554481c3fd1b426b,f611335fc568] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000029, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000023, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:13.323+08:00, sub_task_instance_id=10000029}, listMap={}}, executor=10.120.0.1 2023-11-27 13:36:14.986 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:15.493 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:15.005 WARN 10 --- [pool-101-thread-1,554481c3fd1b426b,cba9b1354064] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:darenOB, ob tenant id:1001 2023-11-27 13:36:15.133 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063375, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:15.138 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063375, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:15.149 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063375, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:15.150 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063375, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:15.150 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063375, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:15.237 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000031, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:15.501 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2023-11-27 13:36:15.517 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20231117 2023-11-27 13:36:15.523 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:15.523 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:15.528 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` drop partition (P20231117) 2023-11-27 13:36:15.562 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000018, status:SUCCESS 2023-11-27 13:36:15.563 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000018, status:SUCCESS 2023-11-27 13:36:15.298 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.ChannelAggregator : aggregate, pre groupAggregator is null, reset it, channelId=100000, groupByKey=0:no_enough_exporter:1, alarmId=2000001, alarmType=no_enough_exporter, alarmTarget=alarm_template_id=0:service=OCP:host=133.197.204.10 2023-11-27 13:36:15.311 INFO 10 --- [pool-subtask-executor19,554481c3fd1b426b,28ee9ff7093d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000030, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000024, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:13.620+08:00, sub_task_instance_id=10000030}, listMap={}}, executor=10.120.0.1 2023-11-27 13:36:15.752 INFO 10 --- [pool-subtask-executor19,554481c3fd1b426b,28ee9ff7093d] c.o.o.b.i.operation.BackupCleanService : create clean backup agent log task: 1701063375752 2023-11-27 13:36:15.624 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.ChannelAggregator : aggregate, pre groupAggregator is null, reset it, channelId=100001, groupByKey=0:no_enough_exporter:1, alarmId=2000001, alarmType=no_enough_exporter, alarmTarget=alarm_template_id=0:service=OCP:host=133.197.204.10 2023-11-27 13:36:15.721 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000031, name=Collect backup files size, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.BackupCollectSizeTask, seriesId=0, startTime=2023-11-27T13:36:15.097+08:00, endTime=null} 2023-11-27 13:36:15.776 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:36:15.865 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:15.926 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:15.929 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:36:12.981992+08:00 2023-11-27 13:36:15.932 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2023-11-27 13:36:15.932 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:36:15.945 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20231118 2023-11-27 13:36:15.955 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:15.955 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:15.958 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` drop partition (P20231118) 2023-11-27 13:36:15.987 INFO 10 --- [pool-subtask-executor18,554481c3fd1b426b,f611335fc568] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000029, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:16.001 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000016, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:16.004 INFO 10 --- [pool-subtask-executor19,554481c3fd1b426b,28ee9ff7093d] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000030, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:16.004 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.s.OcpAlarmInspectService : Archive expired alarm history data: archive event_record success, affectRows=1976 2023-11-27 13:36:16.010 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.s.OcpAlarmInspectService : archive event record count 1976 2023-11-27 13:36:16.031 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.s.OcpAlarmInspectService : skipped due no expired event_record found 2023-11-27 13:36:16.031 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.s.OcpAlarmInspectService : archive event record count 0 2023-11-27 13:36:16.031 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.s.OcpAlarmInspectService : archiveExpiredEventRecord count 1976 2023-11-27 13:36:16.036 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000016, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-11-27T13:36:15.992+08:00, endTime=null} 2023-11-27 13:36:16.092 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000017, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:16.107 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:16.107 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:16.108 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2023-11-27 13:36:16.109 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000017, name=Sync ob wait event, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.ash.internal.task.SyncObWaitEventTask, seriesId=0, startTime=2023-11-27T13:36:16.086+08:00, endTime=null} 2023-11-27 13:36:16.124 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:16.125 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:16.129 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:16.136 INFO 10 --- [pool-subtask-executor20,554481c3fd1b426b,efaef355030d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000031, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000025, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:15.094+08:00, sub_task_instance_id=10000031}, listMap={}}, executor=10.120.0.1 2023-11-27 13:36:16.138 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.s.OcpAlarmInspectService : skipped due no expired event_notification found 2023-11-27 13:36:16.159 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.s.OcpAlarmInspectService : archive event notification count 0 2023-11-27 13:36:16.159 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.s.OcpAlarmInspectService : archiveExpiredEventNotification count 0 2023-11-27 13:36:16.143 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:16.159 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:16.160 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2023-11-27 13:36:16.165 INFO 10 --- [pool-subtask-executor20,554481c3fd1b426b,efaef355030d] c.o.o.b.i.m.ProxyBackupStorageMonitor : collect backup storage info for clusters: 2023-11-27 13:36:16.175 WARN 10 --- [pool-101-thread-1,554481c3fd1b426b,cba9b1354064] o.h.engine.jdbc.spi.SqlExceptionHelper : SQL Error: 1526, SQLState: HY000 2023-11-27 13:36:16.209 ERROR 10 --- [pool-101-thread-1,554481c3fd1b426b,cba9b1354064] o.h.engine.jdbc.spi.SqlExceptionHelper : (conn=185) Table has no partition for value 2023-11-27 13:36:16.211 INFO 10 --- [pool-subtask-executor22,554481c3fd1b426b,339aa78c4266] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000017, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000011, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:16.085+08:00, sub_task_instance_id=10000017}, listMap={cluster_ids=[2, 1]}}, executor=10.120.0.1 2023-11-27 13:36:16.211 INFO 10 --- [pool-subtask-executor20,554481c3fd1b426b,efaef355030d] c.o.o.b.i.m.ProxyBackupStorageMonitor : collectInfoList=[] 2023-11-27 13:36:16.227 INFO 10 --- [pool-subtask-executor20,554481c3fd1b426b,efaef355030d] c.o.o.b.i.m.s.BackupCollectInfoProvider : storageCollectInfoList size:[0] 2023-11-27 13:36:16.235 INFO 10 --- [pool-subtask-executor20,554481c3fd1b426b,efaef355030d] c.o.o.b.i.m.s.StorageCapacityExporter : start pre hook computers 2023-11-27 13:36:16.255 INFO 10 --- [pool-subtask-executor20,554481c3fd1b426b,efaef355030d] c.o.o.b.i.m.s.StorageCapacityExporter : start post hook computers 2023-11-27 13:36:16.217 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000018, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:16.224 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000016, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=8000010, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:15.988+08:00, sub_task_instance_id=10000016}, listMap={cluster_ids=[1, 2]}}, executor=10.120.0.1 2023-11-27 13:36:16.240 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:16.307 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:16.249 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:36:16.288 WARN 10 --- [pool-subtask-executor14,554481c3fd1b426b,c5ea75c89d96] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:org.springframework.dao.DataAccessResourceFailureException: could not execute statement; nested exception is org.hibernate.exception.JDBCConnectionException: could not execute statement, cause:{} org.springframework.dao.DataAccessResourceFailureException: could not execute statement; nested exception is org.hibernate.exception.JDBCConnectionException: could not execute statement at org.springframework.orm.jpa.vendor.HibernateJpaDialect.convertHibernateAccessException(HibernateJpaDialect.java:255) ~[spring-orm-5.3.23.jar!/:5.3.23] at org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:233) ~[spring-orm-5.3.23.jar!/:5.3.23] at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.translateExceptionIfPossible(AbstractEntityManagerFactoryBean.java:551) ~[spring-orm-5.3.23.jar!/:5.3.23] at org.springframework.dao.support.ChainedPersistenceExceptionTranslator.translateExceptionIfPossible(ChainedPersistenceExceptionTranslator.java:61) ~[spring-tx-5.3.23.jar!/:5.3.23] at org.springframework.dao.support.DataAccessUtils.translateIfNecessary(DataAccessUtils.java:242) ~[spring-tx-5.3.23.jar!/:5.3.23] at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:152) ~[spring-tx-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:174) ~[spring-data-jpa-2.6.9.jar!/:2.6.9] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) ~[spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) ~[spring-aop-5.3.23.jar!/:5.3.23] at com.sun.proxy.$Proxy556.saveAll(Unknown Source) ~[na:na] at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:153) ~[ocp-obops-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() ~[ocp-obops-service-4.0.3-20230301.jar!/:4.0.3-20230301] at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) ~[spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) ~[spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) ~[spring-tx-5.3.23.jar!/:5.3.23] at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) ~[spring-tx-5.3.23.jar!/:5.3.23] at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) ~[spring-tx-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) ~[spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) ~[spring-aop-5.3.23.jar!/:5.3.23] at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$a67aeb86.collectMiniCompaction() ~[ocp-obops-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:57) ~[ocp-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.hibernate.exception.JDBCConnectionException: could not execute statement at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:48) ~[hibernate-core-5.6.12.Final.jar!/:5.6.12.Final] at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:37) ~[hibernate-core-5.6.12.Final.jar!/:5.6.12.Final] at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113) ~[hibernate-core-5.6.12.Final.jar!/:5.6.12.Final] at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99) ~[hibernate-core-5.6.12.Final.jar!/:5.6.12.Final] at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:200) ~[hibernate-core-5.6.12.Final.jar!/:5.6.12.Final] at org.hibernate.dialect.identity.GetGeneratedKeysDelegate.executeAndExtract(GetGeneratedKeysDelegate.java:58) ~[hibernate-core-5.6.12.Final.jar!/:5.6.12.Final] at org.hibernate.id.insert.AbstractReturningDelegate.performInsert(AbstractReturningDelegate.java:43) ~[hibernate-core-5.6.12.Final.jar!/:5.6.12.Final] at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3279) ~[hibernate-core-5.6.12.Final.jar!/:5.6.12.Final] at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3885) ~[hibernate-core-5.6.12.Final.jar!/:5.6.12.Final] at org.hibernate.action.internal.EntityIdentityInsertAction.execute(EntityIdentityInsertAction.java:84) ~[hibernate-core-5.6.12.Final.jar!/:5.6.12.Final] at org.hibernate.engine.spi.ActionQueue.execute(ActionQueue.java:645) ~[hibernate-core-5.6.12.Final.jar!/:5.6.12.Final] at org.hibernate.engine.spi.ActionQueue.addResolvedEntityInsertAction(ActionQueue.java:282) ~[hibernate-core-5.6.12.Final.jar!/:5.6.12.Final] at org.hibernate.engine.spi.ActionQueue.addInsertAction(ActionQueue.java:263) ~[hibernate-core-5.6.12.Final.jar!/:5.6.12.Final] at org.hibernate.engine.spi.ActionQueue.addAction(ActionQueue.java:317) ~[hibernate-core-5.6.12.Final.jar!/:5.6.12.Final] at org.hibernate.event.internal.AbstractSaveEventListener.addInsertAction(AbstractSaveEventListener.java:330) ~[hibernate-core-5.6.12.Final.jar!/:5.6.12.Final] at org.hibernate.event.internal.AbstractSaveEventListener.performSaveOrReplicate(AbstractSaveEventListener.java:287) ~[hibernate-core-5.6.12.Final.jar!/:5.6.12.Final] at org.hibernate.event.internal.AbstractSaveEventListener.performSave(AbstractSaveEventListener.java:193) ~[hibernate-core-5.6.12.Final.jar!/:5.6.12.Final] at org.hibernate.event.internal.AbstractSaveEventListener.saveWithGeneratedId(AbstractSaveEventListener.java:123) ~[hibernate-core-5.6.12.Final.jar!/:5.6.12.Final] at org.hibernate.event.internal.DefaultPersistEventListener.entityIsTransient(DefaultPersistEventListener.java:185) ~[hibernate-core-5.6.12.Final.jar!/:5.6.12.Final] at org.hibernate.event.internal.DefaultPersistEventListener.onPersist(DefaultPersistEventListener.java:128) ~[hibernate-core-5.6.12.Final.jar!/:5.6.12.Final] at org.hibernate.event.internal.DefaultPersistEventListener.onPersist(DefaultPersistEventListener.java:55) ~[hibernate-core-5.6.12.Final.jar!/:5.6.12.Final] at org.hibernate.event.service.internal.EventListenerGroupImpl.fireEventOnEachListener(EventListenerGroupImpl.java:107) ~[hibernate-core-5.6.12.Final.jar!/:5.6.12.Final] at org.hibernate.internal.SessionImpl.firePersist(SessionImpl.java:756) ~[hibernate-core-5.6.12.Final.jar!/:5.6.12.Final] at org.hibernate.internal.SessionImpl.persist(SessionImpl.java:742) ~[hibernate-core-5.6.12.Final.jar!/:5.6.12.Final] at sun.reflect.GeneratedMethodAccessor696.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_312] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_312] at org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:311) ~[spring-orm-5.3.23.jar!/:5.3.23] at com.sun.proxy.$Proxy293.persist(Unknown Source) ~[na:na] at org.springframework.data.jpa.repository.support.SimpleJpaRepository.save(SimpleJpaRepository.java:647) ~[spring-data-jpa-2.6.9.jar!/:2.6.9] at org.springframework.data.jpa.repository.support.SimpleJpaRepository.saveAll(SimpleJpaRepository.java:681) ~[spring-data-jpa-2.6.9.jar!/:2.6.9] at org.springframework.data.jpa.repository.support.SimpleJpaRepository.saveAll(SimpleJpaRepository.java:87) ~[spring-data-jpa-2.6.9.jar!/:2.6.9] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_312] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_312] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_312] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_312] at org.springframework.data.repository.core.support.RepositoryMethodInvoker$RepositoryFragmentMethodInvoker.lambda$new$0(RepositoryMethodInvoker.java:289) ~[spring-data-commons-2.6.9.jar!/:2.6.9] at org.springframework.data.repository.core.support.RepositoryMethodInvoker.doInvoke(RepositoryMethodInvoker.java:137) ~[spring-data-commons-2.6.9.jar!/:2.6.9] at org.springframework.data.repository.core.support.RepositoryMethodInvoker.invoke(RepositoryMethodInvoker.java:121) ~[spring-data-commons-2.6.9.jar!/:2.6.9] at org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments.invoke(RepositoryComposition.java:530) ~[spring-data-commons-2.6.9.jar!/:2.6.9] at org.springframework.data.repository.core.support.RepositoryComposition.invoke(RepositoryComposition.java:286) ~[spring-data-commons-2.6.9.jar!/:2.6.9] at org.springframework.data.repository.core.support.RepositoryFactorySupport$ImplementationMethodExecutionInterceptor.invoke(RepositoryFactorySupport.java:640) ~[spring-data-commons-2.6.9.jar!/:2.6.9] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:164) ~[spring-data-commons-2.6.9.jar!/:2.6.9] at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.invoke(QueryExecutorMethodInterceptor.java:139) ~[spring-data-commons-2.6.9.jar!/:2.6.9] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:81) ~[spring-data-commons-2.6.9.jar!/:2.6.9] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) ~[spring-tx-5.3.23.jar!/:5.3.23] at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) ~[spring-tx-5.3.23.jar!/:5.3.23] at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) ~[spring-tx-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137) ~[spring-tx-5.3.23.jar!/:5.3.23] ... 28 common frames omitted Caused by: java.sql.SQLTransientConnectionException: (conn=185) Table has no partition for value at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:202) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseStatement.executeExceptionEpilogue(OceanBaseStatement.java:315) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:243) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.JDBC4PreparedStatement.execute(JDBC4PreparedStatement.java:161) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.JDBC4PreparedStatement.executeUpdate(JDBC4PreparedStatement.java:195) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidPooledPreparedStatement.executeUpdate(DruidPooledPreparedStatement.java:255) ~[druid-1.2.8.jar!/:1.2.8] at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:197) ~[hibernate-core-5.6.12.Final.jar!/:5.6.12.Final] ... 76 common frames omitted Caused by: com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException: Table has no partition for value at com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException.of(OceanBaseSqlException.java:79) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:199) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:180) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:372) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:234) ~[oceanbase-client-2.4.2.jar!/:na] ... 80 common frames omitted Caused by: java.sql.SQLException: Table has no partition for value at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:2192) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2057) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:1951) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:370) ~[oceanbase-client-2.4.2.jar!/:na] ... 81 common frames omitted 2023-11-27 13:36:16.307 INFO 10 --- [pool-subtask-executor14,554481c3fd1b426b,c5ea75c89d96] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2023-11-27 13:36:16.317 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:16.342 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000018, name=Clean expired health check record, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.operation.healthcheck.task.CleanExpiredHealthCheckRecordTask, seriesId=0, startTime=2023-11-27T13:36:16.212+08:00, endTime=null} 2023-11-27 13:36:16.342 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:16.343 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:16.351 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2023-11-27 13:36:16.373 INFO 10 --- [pool-subtask-executor20,554481c3fd1b426b,efaef355030d] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000031, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:16.377 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:16.377 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:16.379 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:16.380 INFO 10 --- [pool-subtask-executor14,554481c3fd1b426b,c5ea75c89d96] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000025, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:16.382 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:16.382 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:16.389 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2023-11-27 13:36:16.439 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:16.439 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:16.447 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:16.454 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:16.455 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:16.459 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2023-11-27 13:36:16.504 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000019, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:16.511 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:16.511 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:16.512 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:16.514 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:16.514 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:16.516 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2023-11-27 13:36:16.550 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000019, name=Check root service , state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CheckClusterRootServersTask, seriesId=0, startTime=2023-11-27T13:36:16.493+08:00, endTime=null} 2023-11-27 13:36:16.558 INFO 10 --- [pool-subtask-executor22,554481c3fd1b426b,339aa78c4266] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:16.575 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:16.575 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:16.580 INFO 10 --- [pool-subtask-executor22,554481c3fd1b426b,339aa78c4266] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:16.580 INFO 10 --- [pool-subtask-executor22,554481c3fd1b426b,339aa78c4266] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:16.588 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:16.599 INFO 10 --- [pool-subtask-executor22,554481c3fd1b426b,339aa78c4266] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:16.610 INFO 10 --- [pool-subtask-executor22,554481c3fd1b426b,339aa78c4266] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:16.610 INFO 10 --- [pool-subtask-executor22,554481c3fd1b426b,339aa78c4266] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:16.613 INFO 10 --- [pool-subtask-executor22,554481c3fd1b426b,339aa78c4266] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select event_id, wait_class as wait_class_name, wait_class_id, event as event_name from v$session_event where sid = (select sid from v$session_event limit 1) 2023-11-27 13:36:16.629 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:16.629 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:16.637 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2023-11-27 13:36:16.671 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:36:16.695 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:16.695 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:16.700 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:16.705 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:16.705 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:16.711 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2023-11-27 13:36:16.720 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000020, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:16.752 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:16.752 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:16.754 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:16.756 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:16.756 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:16.757 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2023-11-27 13:36:16.765 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000020, name=Sql Parse Task, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=2023-11-27T13:36:16.660+08:00, endTime=null} 2023-11-27 13:36:16.767 INFO 10 --- [pool-subtask-executor23,554481c3fd1b426b,df92d3fa46c9] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000018, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000012, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:16.208+08:00, sub_task_instance_id=10000018}, listMap={}}, executor=10.120.0.1 2023-11-27 13:36:16.795 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:16.795 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:16.796 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:16.802 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:16.803 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:16.810 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2023-11-27 13:36:16.834 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:16.834 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:16.838 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:16.845 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:16.845 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:16.846 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2023-11-27 13:36:16.886 INFO 10 --- [pool-subtask-executor24,554481c3fd1b426b,fd23669fc9d4] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000019, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000013, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:16.489+08:00, sub_task_instance_id=10000019}, listMap={}}, executor=10.120.0.1 2023-11-27 13:36:16.894 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:16.894 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:16.904 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:16.914 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:16.914 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:16.915 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2023-11-27 13:36:16.927 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:16.927 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:16.929 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:16.931 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:16.931 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:16.932 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2023-11-27 13:36:16.949 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:16.949 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:16.953 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:16.956 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:16.956 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:16.963 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2023-11-27 13:36:16.972 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000021, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:16.973 INFO 10 --- [pool-subtask-executor24,554481c3fd1b426b,fd23669fc9d4] c.o.o.s.t.b.c.AbstractAllClusterTask : Check root service executed completed, tryCount=2, successCount=0 2023-11-27 13:36:16.999 INFO 10 --- [pool-subtask-executor23,554481c3fd1b426b,df92d3fa46c9] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000018, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:17.002 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:17.002 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:17.004 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:17.006 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:17.006 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:17.008 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2023-11-27 13:36:17.023 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,1bf10434ee48] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000020, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=8000014, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:16.645+08:00, sub_task_instance_id=10000020}, listMap={cluster_ids=[[2,1,1]], tenant_ids=[[5,2,3]]}}, executor=10.120.0.1 2023-11-27 13:36:17.025 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,1bf10434ee48] c.o.o.p.sql.internal.task.SqlParseTask : Parse sql for [5, 2, 3] 2023-11-27 13:36:17.040 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:17.040 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:17.046 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000021, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-11-27T13:36:16.890+08:00, endTime=null} 2023-11-27 13:36:17.048 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:17.051 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:17.051 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:17.053 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2023-11-27 13:36:17.070 INFO 10 --- [pool-subtask-executor24,554481c3fd1b426b,fd23669fc9d4] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000019, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:17.071 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:17.071 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:17.074 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:17.080 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:17.080 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:17.083 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2023-11-27 13:36:17.098 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:17.099 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:17.100 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:17.106 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:17.106 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:17.113 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2023-11-27 13:36:17.149 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:17.149 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:17.150 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:17.153 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:17.153 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:17.154 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2023-11-27 13:36:17.166 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:17.167 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:17.168 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:17.171 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:17.171 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:17.173 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2023-11-27 13:36:17.181 INFO 10 --- [pool-subtask-executor16,554481c3fd1b426b,761a8697ecb7] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_2currentTimeUs:1701063373260000 2023-11-27 13:36:17.181 INFO 10 --- [pool-subtask-executor16,554481c3fd1b426b,761a8697ecb7] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_2diagnosisStartTimeUs:1701061573260000 2023-11-27 13:36:17.184 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:36:17.191 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20231202 2023-11-27 13:36:17.193 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:17.193 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:17.195 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20231202 values less than (1701561600000000)) 2023-11-27 13:36:17.317 INFO 10 --- [ForkJoinPool-14-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:17.320 INFO 10 --- [ForkJoinPool-14-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:17.320 INFO 10 --- [ForkJoinPool-14-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:17.323 INFO 10 --- [ForkJoinPool-14-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:17.333 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000022, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:17.395 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000022, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-11-27T13:36:17.324+08:00, endTime=null} 2023-11-27 13:36:17.504 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:36:17.507 INFO 10 --- [ForkJoinPool-14-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:17.508 INFO 10 --- [ForkJoinPool-14-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:17.508 INFO 10 --- [ForkJoinPool-14-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:17.509 INFO 10 --- [ForkJoinPool-14-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:17.607 INFO 10 --- [ForkJoinPool-14-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:17.607 INFO 10 --- [ForkJoinPool-14-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:17.607 INFO 10 --- [ForkJoinPool-14-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:17.607 INFO 10 --- [ForkJoinPool-14-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:17.608 INFO 10 --- [ForkJoinPool-14-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2023-11-27 13:36:17.612 INFO 10 --- [ForkJoinPool-14-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2023-11-27 13:36:17.612 INFO 10 --- [pool-subtask-executor16,554481c3fd1b426b,761a8697ecb7] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_2totalLength:0 2023-11-27 13:36:17.646 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:17.647 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:17.647 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:17.648 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:17.659 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:17.659 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:17.661 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-11-27 13:36:17.662 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000012, status:SUCCESS 2023-11-27 13:36:17.665 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000012, status:SUCCESS 2023-11-27 13:36:17.676 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:17.676 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:17.677 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-11-27 13:36:17.685 INFO 10 --- [pool-subtask-executor26,554481c3fd1b426b,eca62811bc5b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000021, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000015, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:16.872+08:00, sub_task_instance_id=10000021}, listMap={}}, executor=10.120.0.1 2023-11-27 13:36:17.699 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000023, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:17.744 INFO 10 --- [http-nio-0.0.0.0-8080-exec-10,1a3529a786e04785,3b5e661f93a2] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.10, traceId=1a3529a786e04785, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:36:17.746 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:17.747 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:17.747 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-11-27 13:36:17.801 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:17.801 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:17.807 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-11-27 13:36:17.834 INFO 10 --- [pool-subtask-executor16,554481c3fd1b426b,761a8697ecb7] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_3currentTimeUs:1701063377710000 2023-11-27 13:36:17.834 INFO 10 --- [pool-subtask-executor16,554481c3fd1b426b,761a8697ecb7] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_3diagnosisStartTimeUs:1701061577710000 2023-11-27 13:36:17.904 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:17.904 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:17.910 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-11-27 13:36:17.961 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-11-27 13:36:18.089 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000023, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-11-27T13:36:17.689+08:00, endTime=null} 2023-11-27 13:36:18.110 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=6 2023-11-27 13:36:18.110 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=9 2023-11-27 13:36:18.112 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-11-27 13:36:18.112 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-11-27 13:36:18.147 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:18.147 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:18.147 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2023-11-27 13:36:18.149 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:36:18.201 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:36:18.213 INFO 10 --- [pool-subtask-executor26,554481c3fd1b426b,eca62811bc5b] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000021, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:18.213 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:36:18.215 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:36:18.246 INFO 10 --- [http-nio-0.0.0.0-8080-exec-10,1a3529a786e04785,3b5e661f93a2] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.10, traceId=1a3529a786e04785, duration=504 ms] 2023-11-27 13:36:18.249 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2023-11-27 13:36:18.251 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000013, status:SUCCESS 2023-11-27 13:36:18.292 INFO 10 --- [pool-subtask-executor16,554481c3fd1b426b,761a8697ecb7] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_3totalLength:0 2023-11-27 13:36:18.292 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000013, status:SUCCESS 2023-11-27 13:36:18.303 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606211] update tenant status, tenantId=4, oldStatus=UNAVAILABLE, newStatus=NORMAL 2023-11-27 13:36:18.334 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm is revoled, alarmId=38833, status=Inactive, labels=class OcpLabels { {app=OCP, alarm_name=分區創建失敗, app_type=OCP, alarm_name_en_us=partition create failed, alarm_name_zh_cn=分区创建失败, alarm_level=3, alarm_template_id=0, table_name=ob_hist_sql_plan_monitor, alarm_summary_zh_tw=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 分區創建失敗, result=success, alarm_summary_en_us=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor partition create failed, alarm_description_en_us=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor partition create failed partition (P20231202) create failed, result success., alarm_description_zh_cn=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 分区创建失败 分区（P20231202）创建失败，结果：success。, alarm_type=partition_create_failed, datasource=Monitordb, alarm_status=Inactive, alarm_summary_zh_cn=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 分区创建失败, alarm_scope=Service, alarm_target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor, ocp_ip=10.120.0.1, alarm_description_zh_tw=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 分區創建失敗 分區（P20231202）創建失敗，結果：success。, partition_name=P20231202, alarm_evaluation_interval=0, alarm_name_zh_tw=分區創建失敗} } 2023-11-27 13:36:18.335 INFO 10 --- [pool-subtask-executor16,554481c3fd1b426b,761a8697ecb7] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_5currentTimeUs:1701063378311000 2023-11-27 13:36:18.335 INFO 10 --- [pool-subtask-executor16,554481c3fd1b426b,761a8697ecb7] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_5diagnosisStartTimeUs:1701061578311000 2023-11-27 13:36:18.336 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000012, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:18.415 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20231203 2023-11-27 13:36:18.423 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:18.423 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:18.425 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000012, name=Wait node, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=0, startTime=2023-11-27T13:36:18.310+08:00, endTime=null} 2023-11-27 13:36:18.428 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20231203 values less than (1701648000000000)) 2023-11-27 13:36:18.450 INFO 10 --- [pool-subtask-executor27,554481c3fd1b426b,578aed6ef78f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000022, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000016, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:17.323+08:00, sub_task_instance_id=10000022}, listMap={}}, executor=10.120.0.1 2023-11-27 13:36:18.465 INFO 10 --- [pool-subtask-executor22,554481c3fd1b426b,339aa78c4266] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:18.468 INFO 10 --- [pool-subtask-executor22,554481c3fd1b426b,339aa78c4266] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:18.468 INFO 10 --- [pool-subtask-executor22,554481c3fd1b426b,339aa78c4266] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:18.468 INFO 10 --- [pool-subtask-executor22,554481c3fd1b426b,339aa78c4266] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:18.471 INFO 10 --- [pool-subtask-executor22,554481c3fd1b426b,339aa78c4266] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:18.471 INFO 10 --- [pool-subtask-executor22,554481c3fd1b426b,339aa78c4266] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:18.472 INFO 10 --- [pool-subtask-executor22,554481c3fd1b426b,339aa78c4266] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select event_id, wait_class as wait_class_name, wait_class_id, event as event_name from v$session_event where sid = (select sid from v$session_event limit 1) 2023-11-27 13:36:18.650 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:36:18.675 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000015, status:SUCCESS 2023-11-27 13:36:18.688 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000015, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:18.694 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000015, status:SUCCESS 2023-11-27 13:36:18.702 INFO 10 --- [pool-subtask-executor28,554481c3fd1b426b,0d878e1aa318] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000023, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000017, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:17.689+08:00, sub_task_instance_id=10000023}, listMap={}}, executor=10.120.0.1 2023-11-27 13:36:18.709 INFO 10 --- [pool-subtask-executor16,554481c3fd1b426b,761a8697ecb7] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_5totalLength:0 2023-11-27 13:36:18.720 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=http, hostAddress=133.197.206.92, httpPort=62889, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=basic, basicAuthConfig=BasicAuthConfig(username=ocp_agent), digestAuthConfig=null, formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), posConfig=null, proxy=null), estimated size:9 2023-11-27 13:36:18.721 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.92:62889/api/v1/log/alarms, params:null 2023-11-27 13:36:18.752 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=http, hostAddress=133.197.204.12, httpPort=62889, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=basic, basicAuthConfig=BasicAuthConfig(username=ocp_agent), digestAuthConfig=null, formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), posConfig=null, proxy=null), estimated size:10 2023-11-27 13:36:18.752 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.12:62889/api/v1/log/alarms, params:null 2023-11-27 13:36:18.764 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000015, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-11-27T13:36:18.663+08:00, endTime=null} 2023-11-27 13:36:18.768 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:18.769 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:18.771 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:18.771 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:18.776 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:18.781 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=http, hostAddress=133.197.206.88, httpPort=62889, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=basic, basicAuthConfig=BasicAuthConfig(username=ocp_agent), digestAuthConfig=null, formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), posConfig=null, proxy=null), estimated size:11 2023-11-27 13:36:18.783 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.88:62889/api/v1/log/alarms, params:null 2023-11-27 13:36:18.787 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:18.787 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:18.788 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-11-27 13:36:18.834 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606212] update tenant status by job, tenantId=4, job=TenantJobProgress(tenantId=1, jobId=10, jobType=ALTER_TENANT_LOCALITY, jobStatus=SUCCESS, progress=100), oldStatus=UNAVAILABLE, newStatus=NORMAL 2023-11-27 13:36:18.841 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000019, status:SUCCESS 2023-11-27 13:36:18.843 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=http, hostAddress=133.197.204.3, httpPort=62889, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=basic, basicAuthConfig=BasicAuthConfig(username=ocp_agent), digestAuthConfig=null, formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), posConfig=null, proxy=null), estimated size:12 2023-11-27 13:36:18.843 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.3:62889/api/v1/log/alarms, params:null 2023-11-27 13:36:18.844 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000019, status:SUCCESS 2023-11-27 13:36:18.845 INFO 10 --- [pool-subtask-executor16,554481c3fd1b426b,761a8697ecb7] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000027, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:18.906 INFO 10 --- [pool-subtask-executor29,554481c3fd1b426b,98fb1f7687f7] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000012, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000009, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:18.308+08:00, sub_task_instance_id=10000012}, listMap={cluster_ids=[1, 2]}}, executor=10.120.0.1 2023-11-27 13:36:18.934 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:36:18.939 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:36:18.939 INFO 10 --- [pool-subtask-executor27,554481c3fd1b426b,578aed6ef78f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-11-27 13:36:18.946 INFO 10 --- [pool-subtask-executor27,554481c3fd1b426b,578aed6ef78f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster darenOB, trans stat result is empty. 2023-11-27 13:36:18.955 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000021, status:SUCCESS 2023-11-27 13:36:18.970 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000021, status:SUCCESS 2023-11-27 13:36:19.031 INFO 10 --- [pool-subtask-executor29,554481c3fd1b426b,98fb1f7687f7] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000012, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:19.035 INFO 10 --- [pool-subtask-executor27,554481c3fd1b426b,578aed6ef78f] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000022, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:19.036 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:36:19.145 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-11-27 13:36:19.165 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-11-27 13:36:19.179 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000015, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=8000010, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:18.661+08:00, sub_task_instance_id=10000015}, listMap={cluster_ids=[1, 2]}}, executor=10.120.0.1 2023-11-27 13:36:19.252 INFO 10 --- [pool-subtask-executor28,554481c3fd1b426b,0d878e1aa318] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obcluster, sql count is 0. 2023-11-27 13:36:19.252 INFO 10 --- [pool-subtask-executor28,554481c3fd1b426b,0d878e1aa318] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, result is empty. 2023-11-27 13:36:19.276 INFO 10 --- [pool-subtask-executor28,554481c3fd1b426b,0d878e1aa318] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster darenOB, sql count is 0. 2023-11-27 13:36:19.276 INFO 10 --- [pool-subtask-executor28,554481c3fd1b426b,0d878e1aa318] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster darenOB, result is empty. 2023-11-27 13:36:19.320 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:36:19.326 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-11-27 13:36:19.337 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:19.341 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:19.343 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:19.343 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:19.343 INFO 10 --- [pool-subtask-executor22,554481c3fd1b426b,339aa78c4266] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000017, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:19.345 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:19.349 INFO 10 --- [pool-subtask-executor28,554481c3fd1b426b,0d878e1aa318] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000023, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:19.350 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:19.350 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:19.352 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:36:19.371 INFO 10 --- [http-nio-0.0.0.0-8080-exec-3,e9d3435442714687,e288b1af37e2] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=obcluster&ObClusterId=1&version=2, client=133.197.206.97, traceId=e9d3435442714687, method=OcpApiRtnObj com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obcluster,1,false,2,] 2023-11-27 13:36:19.377 INFO 10 --- [http-nio-0.0.0.0-8080-exec-2,4bbc077f36544a8a,891429e6f0c6] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=darenOB&ObClusterId=10&version=2, client=133.197.206.97, traceId=4bbc077f36544a8a, method=OcpApiRtnObj com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=darenOB,10,false,2,] 2023-11-27 13:36:19.474 INFO 10 --- [http-nio-0.0.0.0-8080-exec-3,e9d3435442714687,e288b1af37e2] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.206.97, traceId=e9d3435442714687, duration=109 ms] 2023-11-27 13:36:19.474 INFO 10 --- [http-nio-0.0.0.0-8080-exec-2,4bbc077f36544a8a,891429e6f0c6] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.206.97, traceId=4bbc077f36544a8a, duration=98 ms] 2023-11-27 13:36:19.488 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] slow query, durationMillis=1060, sql=alter table `ob_hist_sql_plan_monitor` add partition(partition P20231203 values less than (1701648000000000)) 2023-11-27 13:36:19.492 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2023-11-27 13:36:19.492 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:19.492 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:19.525 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2023-11-27 13:36:19.536 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000023, status:SUCCESS 2023-11-27 13:36:19.541 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000023, status:SUCCESS 2023-11-27 13:36:19.546 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2023-11-27 13:36:19.615 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:36:19.618 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606211] update tenant status, tenantId=5, oldStatus=UNAVAILABLE, newStatus=NORMAL 2023-11-27 13:36:19.660 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=http, hostAddress=133.197.204.11, httpPort=62889, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=basic, basicAuthConfig=BasicAuthConfig(username=ocp_agent), digestAuthConfig=null, formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), posConfig=null, proxy=null), estimated size:13 2023-11-27 13:36:19.660 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.11:62889/api/v1/log/alarms, params:null 2023-11-27 13:36:19.685 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm is revoled, alarmId=38833, status=Inactive, labels=class OcpLabels { {app=OCP, alarm_name=分區創建失敗, app_type=OCP, alarm_name_en_us=partition create failed, alarm_name_zh_cn=分区创建失败, alarm_level=3, alarm_template_id=0, table_name=ob_hist_sql_plan_monitor, alarm_summary_zh_tw=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 分區創建失敗, result=success, alarm_summary_en_us=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor partition create failed, alarm_description_en_us=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor partition create failed partition (P20231203) create failed, result success., alarm_description_zh_cn=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 分区创建失败 分区（P20231203）创建失败，结果：success。, alarm_type=partition_create_failed, datasource=Monitordb, alarm_status=Inactive, alarm_summary_zh_cn=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 分区创建失败, alarm_scope=Service, alarm_target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor, ocp_ip=10.120.0.1, alarm_description_zh_tw=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 分區創建失敗 分區（P20231203）創建失敗，結果：success。, partition_name=P20231203, alarm_evaluation_interval=0, alarm_name_zh_tw=分區創建失敗} } 2023-11-27 13:36:19.725 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=http, hostAddress=133.197.206.97, httpPort=62889, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=basic, basicAuthConfig=BasicAuthConfig(username=ocp_agent), digestAuthConfig=null, formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), posConfig=null, proxy=null), estimated size:14 2023-11-27 13:36:19.725 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.97:62889/api/v1/log/alarms, params:null 2023-11-27 13:36:19.739 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20231204 2023-11-27 13:36:19.765 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:19.766 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:19.779 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000024, status:SUCCESS 2023-11-27 13:36:19.780 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20231204 values less than (1701734400000000)) 2023-11-27 13:36:19.783 INFO 10 --- [ForkJoinPool-14-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:19.784 INFO 10 --- [ForkJoinPool-14-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:19.798 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000024, status:SUCCESS 2023-11-27 13:36:19.801 INFO 10 --- [ForkJoinPool-14-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:19.802 INFO 10 --- [ForkJoinPool-14-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:19.802 INFO 10 --- [ForkJoinPool-14-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2023-11-27 13:36:19.812 INFO 10 --- [ForkJoinPool-14-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:19.812 INFO 10 --- [ForkJoinPool-14-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:19.821 INFO 10 --- [ForkJoinPool-14-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2023-11-27 13:36:19.821 INFO 10 --- [ForkJoinPool-14-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, stop_time, start_service_time, block_migrate_in_time FROM oceanbase.__all_server where svr_ip = ? and svr_port = ?, args: [133.197.206.97, 2882] 2023-11-27 13:36:19.872 INFO 10 --- [ForkJoinPool-14-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:19.872 INFO 10 --- [ForkJoinPool-14-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:19.873 INFO 10 --- [ForkJoinPool-14-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [133.197.204.10, 2882] 2023-11-27 13:36:19.879 INFO 10 --- [ForkJoinPool-14-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:19.879 INFO 10 --- [ForkJoinPool-14-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:19.882 INFO 10 --- [ForkJoinPool-14-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [133.197.204.11, 2882] 2023-11-27 13:36:19.886 INFO 10 --- [ForkJoinPool-14-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:19.886 INFO 10 --- [ForkJoinPool-14-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:19.888 INFO 10 --- [ForkJoinPool-14-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [133.197.204.12, 2882] 2023-11-27 13:36:19.888 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=http, hostAddress=133.197.204.13, httpPort=62889, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=basic, basicAuthConfig=BasicAuthConfig(username=ocp_agent), digestAuthConfig=null, formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), posConfig=null, proxy=null), estimated size:15 2023-11-27 13:36:19.888 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.13:62889/api/v1/log/alarms, params:null 2023-11-27 13:36:19.892 INFO 10 --- [ForkJoinPool-14-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:19.892 INFO 10 --- [ForkJoinPool-14-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:19.894 INFO 10 --- [ForkJoinPool-14-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [133.197.204.13, 2882] 2023-11-27 13:36:19.903 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:19.905 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:19.909 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:19.909 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:19.911 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:19.915 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:19.916 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:19.917 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-11-27 13:36:19.919 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606212] update tenant status by job, tenantId=5, job=TenantJobProgress(tenantId=1002, jobId=9, jobType=ALTER_TENANT_LOCALITY, jobStatus=SUCCESS, progress=100), oldStatus=UNAVAILABLE, newStatus=NORMAL 2023-11-27 13:36:19.985 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:36:20.066 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000006, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:20.126 INFO 10 --- [http-nio-0.0.0.0-8080-exec-4,e33e5afda1214e18,8fc5abcd6370] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.10, traceId=e33e5afda1214e18, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:36:20.137 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063380, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:20.140 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063380, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:20.140 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063380, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:20.144 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000006, name=Generate cluster snapshot, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.diag.internal.task.GenerateSnapshotTask, seriesId=1, startTime=2023-11-27T13:36:20.035+08:00, endTime=null} 2023-11-27 13:36:20.152 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000025, status:SUCCESS 2023-11-27 13:36:20.187 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063380, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:20.198 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000025, status:SUCCESS 2023-11-27 13:36:20.219 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063380, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:20.321 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000007, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:20.323 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:20.325 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:20.328 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:20.329 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:20.330 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:20.336 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:20.336 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:20.337 INFO 10 --- [http-nio-0.0.0.0-8080-exec-4,e33e5afda1214e18,8fc5abcd6370] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.10, traceId=e33e5afda1214e18, duration=211 ms] 2023-11-27 13:36:20.338 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:36:20.339 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000007, name=Generate cluster snapshot, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.diag.internal.task.GenerateSnapshotTask, seriesId=2, startTime=2023-11-27T13:36:20.259+08:00, endTime=null} 2023-11-27 13:36:20.340 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-11-27 13:36:20.351 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:20.368 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:36:20.379 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:20.379 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:20.395 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:20.405 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000006, context=Context{parallelIdx=0, stringMap={begin_interval_time=2023-11-27T12:36:06.973+08:00, prohibit_rollback=false, task_instance_id=8000005, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:20.032+08:00, end_interval_time=2023-11-27T13:36:06.973+08:00, sub_task_instance_id=10000006}, listMap={cluster_ids=[1, 2]}}, executor=10.120.0.1 2023-11-27 13:36:20.411 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:20.412 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:20.419 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, tenant_name, (CASE WHEN tenant_id = 1 THEN 'SYS' ELSE 'USER' END) AS tenant_type, (CASE WHEN compatibility_mode = 1 THEN 'ORACLE' ELSE 'MYSQL' END) AS compatibility_mode, zone_list, primary_zone, locality, status, locked, read_only, in_recyclebin, TIMESTAMPDIFF(SECOND, gmt_create, now()) AS exist_seconds FROM oceanbase.__all_tenant 2023-11-27 13:36:20.434 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:20.434 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:20.446 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1] 2023-11-27 13:36:20.512 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:20.512 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:20.535 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1] 2023-11-27 13:36:20.572 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.120.0.1 2023-11-27 13:36:20.572 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:20.573 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:20.594 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1001] 2023-11-27 13:36:20.609 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000007, context=Context{parallelIdx=1, stringMap={begin_interval_time=2023-11-27T12:36:06.973+08:00, prohibit_rollback=false, task_instance_id=8000005, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:20.249+08:00, end_interval_time=2023-11-27T13:36:06.973+08:00, sub_task_instance_id=10000007}, listMap={cluster_ids=[1, 2]}}, executor=10.120.0.1 2023-11-27 13:36:20.619 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:20.619 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:20.640 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2023-11-27 13:36:20.791 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm is revoled, alarmId=38833, status=Inactive, labels=class OcpLabels { {app=OCP, alarm_name=分區創建失敗, app_type=OCP, alarm_name_en_us=partition create failed, alarm_name_zh_cn=分区创建失败, alarm_level=3, alarm_template_id=0, table_name=ob_hist_sql_plan_monitor, alarm_summary_zh_tw=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 分區創建失敗, result=success, alarm_summary_en_us=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor partition create failed, alarm_description_en_us=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor partition create failed partition (P20231204) create failed, result success., alarm_description_zh_cn=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 分区创建失败 分区（P20231204）创建失败，结果：success。, alarm_type=partition_create_failed, datasource=Monitordb, alarm_status=Inactive, alarm_summary_zh_cn=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 分区创建失败, alarm_scope=Service, alarm_target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor, ocp_ip=10.120.0.1, alarm_description_zh_tw=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 分區創建失敗 分區（P20231204）創建失敗，結果：success。, partition_name=P20231204, alarm_evaluation_interval=0, alarm_name_zh_tw=分區創建失敗} } 2023-11-27 13:36:20.792 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:20.792 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:20.798 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1001] 2023-11-27 13:36:20.819 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:20.819 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:20.820 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1002] 2023-11-27 13:36:20.841 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:20.841 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:20.842 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1002] 2023-11-27 13:36:20.853 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-11-27 13:36:20.865 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20231109 2023-11-27 13:36:20.867 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:20.867 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:20.868 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` drop partition (P20231109) 2023-11-27 13:36:21.157 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:36:21.229 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:21.229 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:21.229 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2023-11-27 13:36:21.239 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20231110 2023-11-27 13:36:21.240 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:21.240 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:21.240 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` drop partition (P20231110) 2023-11-27 13:36:21.251 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:21.254 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:21.261 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:21.262 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:21.264 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:21.586 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:21.586 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:21.588 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2023-11-27 13:36:21.598 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20231111 2023-11-27 13:36:21.599 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:21.601 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:21.601 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:21.602 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` drop partition (P20231111) 2023-11-27 13:36:21.606 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:21.607 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:21.607 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:21.608 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:21.610 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:21.610 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:21.610 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-11-27 13:36:21.939 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:36:21.943 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:21.943 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:21.944 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2023-11-27 13:36:21.954 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:21.954 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:21.955 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:21.957 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:21.957 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:21.957 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat 2023-11-27 13:36:21.971 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat, partition=P20231203 2023-11-27 13:36:21.972 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:21.972 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:21.973 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat` add partition(partition P20231203 values less than (1701648000000000)) 2023-11-27 13:36:21.991 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:21.991 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:21.992 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-11-27 13:36:22.011 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:36:22.039 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : start to generate snapshot for cluster: 1 2023-11-27 13:36:22.040 WARN 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid time to generate snapshot, time error!, current time is 2023-11-27T13:36:22.040+08:00, end interval time is 2023-11-27T13:36:06.973+08:00 2023-11-27 13:36:22.041 WARN 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid server status, some servers have been restarted. server info is [ObServerEntity{id=1, ip=133.197.206.97, port=2882, sqlPort=2881, version=3.1.4_10000092022071511-b4bfa011ceaef428782dcb65ae89190c40b78c2f(Jul 15 2022 11:45:14), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-27T13:30:19+08:00, stopTime=1970-01-01T08:00+08:00}] 2023-11-27 13:36:22.136 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : start to generate snapshot for cluster: 2 2023-11-27 13:36:22.136 WARN 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid time to generate snapshot, time error!, current time is 2023-11-27T13:36:22.136+08:00, end interval time is 2023-11-27T13:36:06.973+08:00 2023-11-27 13:36:22.182 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:36:22.191 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:36:22.266 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : start to produce cluster snapshot data: cluster_id = 2 2023-11-27 13:36:22.267 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot, context=GenerateSnapshotContext{clusterEntity=ObClusterEntity{id=2, name=darenOB, obClusterId=10, status=RUNNING}, snapshotId=8139, beginIntervalTime=1701059766973000, beginIntervalTimeString=2023-11-27T12:36:06.973, endIntervalTime=1701063366973000, endIntervalTimeString=2023-11-27T13:36:06.973} 2023-11-27 13:36:22.269 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for sqlAuditSnapshotOperator 2023-11-27 13:36:22.269 WARN 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid time to generate snapshot data, snapshot end interval time is 1701063366973000, now is 1701063382269000 2023-11-27 13:36:22.337 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-11-27 13:36:22.338 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:22.338 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:22.340 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-11-27 13:36:22.352 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:36:22.366 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : start to produce cluster snapshot data: cluster_id = 1 2023-11-27 13:36:22.366 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot, context=GenerateSnapshotContext{clusterEntity=ObClusterEntity{id=1, name=obcluster, obClusterId=1, status=RUNNING}, snapshotId=8140, beginIntervalTime=1701059766973000, beginIntervalTimeString=2023-11-27T12:36:06.973, endIntervalTime=1701063366973000, endIntervalTimeString=2023-11-27T13:36:06.973} 2023-11-27 13:36:22.367 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for sqlAuditSnapshotOperator 2023-11-27 13:36:22.367 WARN 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid time to generate snapshot data, snapshot end interval time is 1701063366973000, now is 1701063382367000 2023-11-27 13:36:22.406 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:36:07.0, nextRun=2023-11-27 13:36:20.0, currentDate=Mon Nov 27 13:36:22 CST 2023 2023-11-27 13:36:22.406 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-11-27 13:36:07.0, nextRunTime=2023-11-27 13:36:20.0} 2023-11-27 13:36:22.438 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:22.443 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:22.443 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:22.444 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:22.449 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:22.458 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:22.458 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:22.462 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:36:22.471 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-11-27 13:36:22.471 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-11-27 13:36:22.471 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=darenOB, obClusterId=10 2023-11-27 13:36:22.499 INFO 10 --- [pool-overview-container-8,,55968b847e77] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=417 2023-11-27 13:36:22.506 INFO 10 --- [pool-overview-container-8,,55968b847e77] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=11000 2023-11-27 13:36:22.537 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-11-27 13:36:22.537 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:22.537 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:22.921 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:22.921 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:22.922 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat' and table_type='BASE TABLE' 2023-11-27 13:36:22.943 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,dfae5c8cd7f8] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000016, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:23.046 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000032, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:23.049 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:23.050 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:23.050 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:23.050 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:23.050 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:23.052 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:23.052 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:23.053 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1001, ALTER_TENANT_LOCALITY] 2023-11-27 13:36:23.063 INFO 10 --- [pool-scanner--1234-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=132 2023-11-27 13:36:23.083 INFO 10 --- [pool-ocp-schedules-8,,766a8334215e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:36:23.104 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat 2023-11-27 13:36:23.104 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:36:23.110 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat, partition=P20231204 2023-11-27 13:36:23.113 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:23.113 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:23.120 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat` add partition(partition P20231204 values less than (1701734400000000)) 2023-11-27 13:36:23.145 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000032, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-11-27T13:36:22.976+08:00, endTime=null} 2023-11-27 13:36:23.164 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000011, status:SUCCESS 2023-11-27 13:36:23.182 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000011, status:SUCCESS 2023-11-27 13:36:23.186 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:23.192 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:23.192 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:23.196 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:23.199 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:23.207 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:23.207 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:23.213 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:23.216 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:23.216 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:23.218 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, stop_time, start_service_time, block_migrate_in_time, id FROM oceanbase.__all_server ORDER BY zone ASC, id ASC 2023-11-27 13:36:23.219 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:23.219 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:23.221 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.o.SqlAuditStatOperator : Produce base for 1, [Pair(left=Tenant(id=1, name=sys, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1, sqlAuditEnabled=null), right=ObServer(svrIp=133.197.206.97, svrPort=2882, innerPort=2881, withRootserver=true, status=ACTIVE, zone=zone1, buildVersion=3.1.4_10000092022071511-b4bfa011ceaef428782dcb65ae89190c40b78c2f(Jul 15 2022 11:45:14), stopTime=0, startServiceTime=1701063018577428, blockMigrateInTime=0, id=1)), Pair(left=Tenant(id=2, name=meta_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1001, sqlAuditEnabled=null), right=ObServer(svrIp=133.197.206.97, svrPort=2882, innerPort=2881, withRootserver=true, status=ACTIVE, zone=zone1, buildVersion=3.1.4_10000092022071511-b4bfa011ceaef428782dcb65ae89190c40b78c2f(Jul 15 2022 11:45:14), stopTime=0, startServiceTime=1701063018577428, blockMigrateInTime=0, id=1)), Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1002, sqlAuditEnabled=null), right=ObServer(svrIp=133.197.206.97, svrPort=2882, innerPort=2881, withRootserver=true, status=ACTIVE, zone=zone1, buildVersion=3.1.4_10000092022071511-b4bfa011ceaef428782dcb65ae89190c40b78c2f(Jul 15 2022 11:45:14), stopTime=0, startServiceTime=1701063018577428, blockMigrateInTime=0, id=1))], between 1701059766973000:1701063366973000 2023-11-27 13:36:23.236 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.120.0.1:8080 2023-11-27 13:36:23.246 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-11-27 13:36:23.252 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:36:23.250440+08:00, lastProcessTime=2023-11-27T13:36:12.981992+08:00 2023-11-27 13:36:23.252 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.o.SqlAuditStatOperator : Produce base for 2, [Pair(left=Tenant(id=4, name=sys, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1;zone3;zone4;zone5;zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=ObServer(svrIp=133.197.204.3, svrPort=2882, innerPort=2881, withRootserver=false, status=INACTIVE, zone=zone1, buildVersion=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), stopTime=0, startServiceTime=0, blockMigrateInTime=0, id=1)), Pair(left=Tenant(id=4, name=sys, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1;zone3;zone4;zone5;zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=ObServer(svrIp=133.197.204.10, svrPort=2882, innerPort=2881, withRootserver=true, status=ACTIVE, zone=zone3, buildVersion=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), stopTime=0, startServiceTime=1695730092226557, blockMigrateInTime=0, id=3)), Pair(left=Tenant(id=4, name=sys, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1;zone3;zone4;zone5;zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=ObServer(svrIp=133.197.204.11, svrPort=2882, innerPort=2881, withRootserver=false, status=ACTIVE, zone=zone4, buildVersion=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), stopTime=0, startServiceTime=1695730305487530, blockMigrateInTime=0, id=4)), Pair(left=Tenant(id=4, name=sys, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1;zone3;zone4;zone5;zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=ObServer(svrIp=133.197.204.12, svrPort=2882, innerPort=2881, withRootserver=false, status=ACTIVE, zone=zone5, buildVersion=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), stopTime=0, startServiceTime=1695730736949667, blockMigrateInTime=0, id=5)), Pair(left=Tenant(id=4, name=sys, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1;zone3;zone4;zone5;zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=ObServer(svrIp=133.197.204.13, svrPort=2882, innerPort=2881, withRootserver=false, status=ACTIVE, zone=zone6, buildVersion=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), stopTime=0, startServiceTime=1695731005656813, blockMigrateInTime=0, id=6)), Pair(left=Tenant(id=5, name=daren, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1,zone3,zone4,zone5,zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=ObServer(svrIp=133.197.204.3, svrPort=2882, innerPort=2881, withRootserver=false, status=INACTIVE, zone=zone1, buildVersion=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), stopTime=0, startServiceTime=0, blockMigrateInTime=0, id=1)), Pair(left=Tenant(id=5, name=daren, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1,zone3,zone4,zone5,zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=ObServer(svrIp=133.197.204.10, svrPort=2882, innerPort=2881, withRootserver=true, status=ACTIVE, zone=zone3, buildVersion=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), stopTime=0, startServiceTime=1695730092226557, blockMigrateInTime=0, id=3)), Pair(left=Tenant(id=5, name=daren, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1,zone3,zone4,zone5,zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=ObServer(svrIp=133.197.204.11, svrPort=2882, innerPort=2881, withRootserver=false, status=ACTIVE, zone=zone4, buildVersion=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), stopTime=0, startServiceTime=1695730305487530, blockMigrateInTime=0, id=4)), Pair(left=Tenant(id=5, name=daren, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1,zone3,zone4,zone5,zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=ObServer(svrIp=133.197.204.12, svrPort=2882, innerPort=2881, withRootserver=false, status=ACTIVE, zone=zone5, buildVersion=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), stopTime=0, startServiceTime=1695730736949667, blockMigrateInTime=0, id=5)), Pair(left=Tenant(id=5, name=daren, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1,zone3,zone4,zone5,zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=ObServer(svrIp=133.197.204.13, svrPort=2882, innerPort=2881, withRootserver=false, status=ACTIVE, zone=zone6, buildVersion=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), stopTime=0, startServiceTime=1695731005656813, blockMigrateInTime=0, id=6))], between 1701059766973000:1701063366973000 2023-11-27 13:36:23.277 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:23.281 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:23.284 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:23.284 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:23.286 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:23.300 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:23.300 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:23.304 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1001, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:36:23.411 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000016, status:SUCCESS 2023-11-27 13:36:23.428 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000016, status:SUCCESS 2023-11-27 13:36:23.467 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.120.0.1:8080 2023-11-27 13:36:23.538 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:23.538 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:23.546 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat' and table_type='BASE TABLE' 2023-11-27 13:36:23.583 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.120.0.1:8080 2023-11-27 13:36:23.620 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:36:23.622 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.o.SqlAuditStatOperator : Finished Producing base for cluster: 1, between 1701059766973000:1701063366973000 2023-11-27 13:36:23.622 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for sqlAuditSnapshotOperator, error count is 1 2023-11-27 13:36:23.622 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for CacheSnapshotOperator 2023-11-27 13:36:23.622 WARN 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid time to generate snapshot data, snapshot end interval time is 1701063366973000, now is 1701063383622000 2023-11-27 13:36:23.626 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:36:23.674 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.120.0.1:8080 2023-11-27 13:36:23.679 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat 2023-11-27 13:36:23.682 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:23.690 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:23.693 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:23.693 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:23.694 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:23.695 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:23.695 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:23.696 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-11-27 13:36:23.698 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat, partition=P20231205 2023-11-27 13:36:23.703 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-11-27 13:36:23.707 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:23.708 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:23.708 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat` add partition(partition P20231205 values less than (1701820800000000)) 2023-11-27 13:36:23.731 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:23.734 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:23.735 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:23.736 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:23.737 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:23.741 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : groupAggregate, now=2023-11-27T13:36:23.250440+08:00, groupByKey=0:no_enough_exporter:1, alarmsToSentCount=4, alarmIds=[2000004, 2000002, 2000003, 2000001] 2023-11-27 13:36:23.741 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : createNotification start, groupByKey=0:no_enough_exporter:1 2023-11-27 13:36:23.752 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:23.753 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:23.754 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,16d2b612fa8c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000032, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000026, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:22.975+08:00, sub_task_instance_id=10000032}, listMap={}}, executor=10.120.0.1 2023-11-27 13:36:23.773 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:36:23.774 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : createNotification done, groupByKey=0:no_enough_exporter:1, alarmIds=[2000004, 2000002, 2000003, 2000001] 2023-11-27 13:36:23.774 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.ChannelAggregator : aggregate done, channelId=100001, now=2023-11-27T13:36:23.250440+08:00, resultsCount=1 2023-11-27 13:36:23.774 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : aggregate done, channelId=100001, alarmsCount=0, resultsCount=1 2023-11-27 13:36:23.774 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.ChannelAggregator : groupAggregate markSendSuccess, groupByKey=0:no_enough_exporter:1, now=2023-11-27T13:36:23.250440+08:00, alarmIds=[2000004, 2000002, 2000003, 2000001] 2023-11-27 13:36:23.774 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:36:23.250440+08:00 2023-11-27 13:36:23.778 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-11-27 13:36:23.795 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000017, status:SUCCESS 2023-11-27 13:36:23.798 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,16d2b612fa8c] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-11-27 13:36:23.798 INFO 10 --- [ForkJoinPool-17-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-11-27 13:36:23.800 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000017, status:SUCCESS 2023-11-27 13:36:23.810 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.o.SqlAuditStatOperator : Finished Producing base for cluster: 2, between 1701059766973000:1701063366973000 2023-11-27 13:36:23.810 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for sqlAuditSnapshotOperator, error count is 0 2023-11-27 13:36:23.810 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for CacheSnapshotOperator 2023-11-27 13:36:23.811 WARN 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid time to generate snapshot data, snapshot end interval time is 1701063366973000, now is 1701063383811000 2023-11-27 13:36:23.819 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:36:23.832 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-11-27 13:36:23.840 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,16d2b612fa8c] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000032, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:23.855 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.120.0.1 2023-11-27 13:36:23.923 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5] 2023-11-27 13:36:23.923 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5] 2023-11-27 13:36:24.197 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:36:24.230 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:24.231 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:24.232 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:24.234 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:24.235 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:24.235 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:24.235 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:36:24.236 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:24.261 INFO 10 --- [ForkJoinPool-17-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:24.264 INFO 10 --- [ForkJoinPool-17-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:24.264 INFO 10 --- [ForkJoinPool-17-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:24.265 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat' and table_type='BASE TABLE' 2023-11-27 13:36:24.273 INFO 10 --- [ForkJoinPool-17-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:24.326 INFO 10 --- [ForkJoinPool-17-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:24.326 INFO 10 --- [ForkJoinPool-17-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:24.338 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:24.338 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:24.339 INFO 10 --- [ForkJoinPool-17-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT MAX(CASE `name` WHEN 'cluster' THEN `info` ELSE '' END ) `cluster`, MAX(CASE `name` WHEN 'merge_status' THEN `info` ELSE '' END ) `merge_status`, MAX(CASE `name` WHEN 'global_broadcast_version' THEN `value` ELSE 0 END ) `global_broadcast_version`, MAX(CASE `name` WHEN 'frozen_version' THEN `value` ELSE 0 END ) `frozen_version`, MAX(CASE `name` WHEN 'last_merged_version' THEN `value` ELSE 0 END ) `last_merged_version`, MAX(CASE `name` WHEN 'frozen_time' THEN `value` ELSE 0 END ) `frozen_time`, MAX(CASE `name` WHEN 'is_merge_error' THEN `value` ELSE 0 END ) `merge_error` FROM oceanbase.__all_zone WHERE `zone` = '' 2023-11-27 13:36:24.349 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1] 2023-11-27 13:36:24.367 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:36:24.367 INFO 10 --- [ForkJoinPool-17-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:24.367 INFO 10 --- [ForkJoinPool-17-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:24.375 INFO 10 --- [ForkJoinPool-17-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select column_value from oceanbase.__all_core_table where table_name ='__all_global_stat' and column_name = 'frozen_version' 2023-11-27 13:36:24.384 INFO 10 --- [ForkJoinPool-17-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:24.385 INFO 10 --- [ForkJoinPool-17-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:24.388 INFO 10 --- [ForkJoinPool-17-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, MAX(CASE `name` WHEN 'region' THEN `info` ELSE '' END ) `region`, MAX(CASE `name` WHEN 'idc' THEN `info` ELSE '' END ) `idc`, MAX(CASE `name` WHEN 'status' THEN `info` ELSE '' END ) `status`, MAX(CASE `name` WHEN 'merge_status' THEN `info` ELSE '' END ) `merge_status`, MAX(CASE `name` WHEN 'broadcast_version' THEN `value` ELSE 0 END ) `broadcast_version`, MAX(CASE `name` WHEN 'all_merged_version' THEN `value` ELSE 0 END ) `all_merged_version`, MAX(CASE `name` WHEN 'last_merged_version' THEN `value` ELSE 0 END ) `last_merged_version`, MAX(CASE `name` WHEN 'merge_start_time' THEN `value` ELSE 0 END ) `merge_start_time`, MAX(CASE `name` WHEN 'last_merged_time' THEN `value` ELSE 0 END ) `last_merged_time`, MAX(CASE `name` WHEN 'is_merge_timeout' THEN `value` ELSE 0 END ) `merge_timeout` FROM oceanbase.__all_zone WHERE `zone` <> '' GROUP BY `zone` 2023-11-27 13:36:24.394 INFO 10 --- [ForkJoinPool-17-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : compaction collect success, cluster=ObClusterOverview(id=1, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"133.197.206.97:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1700793672036000}, createTime=2022-11-29T14:52:53+08:00, creator=admin, obVersion=3.1.4, tenantCount=3) 2023-11-27 13:36:24.398 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-11-27 13:36:24.398 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:24.399 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:24.400 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1001] 2023-11-27 13:36:24.402 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 1000 2023-11-27 13:36:24.419 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:36:24.443 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-11-27 13:36:24.447 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:24.447 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:24.448 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000013, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:24.454 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1002] 2023-11-27 13:36:24.459 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat 2023-11-27 13:36:24.472 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_trans_stat, partition=P20231117 2023-11-27 13:36:24.478 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:36:24.489 INFO 10 --- [pool-scanner--1237-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:24.497 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:24.498 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:24.500 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat` drop partition (P20231117) 2023-11-27 13:36:24.529 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-11-27 13:36:24.529 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-11-27 13:36:24.530 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=obcluster, obClusterId=1 2023-11-27 13:36:24.535 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000013, name=Sync system event, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask, seriesId=1, startTime=2023-11-27T13:36:24.410+08:00, endTime=null} 2023-11-27 13:36:24.537 INFO 10 --- [pool-scanner--1235-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:24.553 INFO 10 --- [pool-scanner--1236-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:24.554 INFO 10 --- [pool-scanner--1238-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:24.555 INFO 10 --- [pool-scanner--1239-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:24.581 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:24.581 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:24.583 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,b897eef7249c] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000015, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:24.588 INFO 10 --- [pool-scanner--1244-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:24.589 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat' and table_type='BASE TABLE' 2023-11-27 13:36:24.610 INFO 10 --- [pool-scanner--1242-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:24.624 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:36:24.628 INFO 10 --- [pool-scanner--1245-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:24.635 INFO 10 --- [pool-scanner--1241-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:24.678 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_trans_stat, partition=P20231118 2023-11-27 13:36:24.684 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:24.684 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:24.685 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat` drop partition (P20231118) 2023-11-27 13:36:24.687 INFO 10 --- [pool-scanner--1246-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:24.759 INFO 10 --- [pool-scanner--1249-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=24 2023-11-27 13:36:24.769 INFO 10 --- [pool-scanner--1248-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:24.772 INFO 10 --- [pool-scanner--1250-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=24 2023-11-27 13:36:24.775 INFO 10 --- [pool-scanner--1247-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:24.786 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000014, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:24.788 INFO 10 --- [pool-scanner--1251-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:24.791 INFO 10 --- [pool-scanner--1252-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:24.792 INFO 10 --- [pool-scanner--1253-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:24.816 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.service.OcpAlarmSearchService : batch get alarm history ids length: 889 2023-11-27 13:36:24.823 INFO 10 --- [pool-scanner--1254-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:24.825 INFO 10 --- [pool-scanner--1255-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:24.832 INFO 10 --- [pool-scanner--1258-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:24.840 INFO 10 --- [pool-scanner--1256-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:24.851 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:24.851 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:24.852 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000014, name=Sync system event, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask, seriesId=2, startTime=2023-11-27T13:36:24.774+08:00, endTime=null} 2023-11-27 13:36:24.852 INFO 10 --- [pool-scanner--1257-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:24.855 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat' and table_type='BASE TABLE' 2023-11-27 13:36:24.900 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_hist_trans_stat, partition=P20231119 2023-11-27 13:36:24.901 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:24.901 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:24.902 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat` drop partition (P20231119) 2023-11-27 13:36:24.962 INFO 10 --- [pool-scanner--1262-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:24.966 INFO 10 --- [pool-scanner--1259-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:24.983 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,878f2d940ab2] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000013, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=8000009, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:24.405+08:00, sub_task_instance_id=10000013}, listMap={cluster_ids=[1, 2]}}, executor=10.120.0.1 2023-11-27 13:36:24.998 INFO 10 --- [pool-scanner--1261-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:25.037 INFO 10 --- [pool-scanner--1260-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:25.061 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:36:25.061 INFO 10 --- [pool-scanner--1263-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=24 2023-11-27 13:36:25.082 INFO 10 --- [pool-scanner--1264-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=24 2023-11-27 13:36:25.099 INFO 10 --- [pool-scanner--1266-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:25.144 INFO 10 --- [pool-scanner--1265-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:25.151 INFO 10 --- [pool-scanner--1269-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=24 2023-11-27 13:36:25.152 INFO 10 --- [pool-scanner--1268-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:25.191 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:36:25.200 INFO 10 --- [pool-scanner--1271-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:25.206 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:36:25.219 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:25.219 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:25.251 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063385, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:25.263 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063385, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:25.274 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat' and table_type='BASE TABLE' 2023-11-27 13:36:25.275 INFO 10 --- [pool-scanner--1267-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:25.293 INFO 10 --- [pool-scanner--1273-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:25.323 INFO 10 --- [pool-scanner--1270-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:25.323 INFO 10 --- [pool-scanner--1272-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:25.327 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:25.327 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:25.334 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:25.352 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for CacheSnapshotOperator, error count is 0 2023-11-27 13:36:25.352 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for LatchSnapshotOperator 2023-11-27 13:36:25.352 WARN 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid time to generate snapshot data, snapshot end interval time is 1701063366973000, now is 1701063385352000 2023-11-27 13:36:25.354 INFO 10 --- [pool-scanner--1275-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:25.360 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063385, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:25.373 INFO 10 --- [pool-scanner--1276-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:25.375 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:25.375 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:25.378 INFO 10 --- [pool-scanner--1274-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:25.380 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063385, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:25.381 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063385, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:25.394 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2023-11-27 13:36:25.410 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for CacheSnapshotOperator, error count is 1 2023-11-27 13:36:25.410 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for LatchSnapshotOperator 2023-11-27 13:36:25.411 WARN 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid time to generate snapshot data, snapshot end interval time is 1701063366973000, now is 1701063385411000 2023-11-27 13:36:25.416 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.service.OcpAlarmSearchService : total load alarm history size: 15889 2023-11-27 13:36:25.416 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.s.a.service.OcpAlarmSearchService : refresh alarm search done 2023-11-27 13:36:25.446 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1701100800000000 2023-11-27 13:36:25.450 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:25.451 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:25.465 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1701100800000000 values less than (1701100800000000)) 2023-11-27 13:36:25.470 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-11-27 13:36:25.473 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,ceb110763ffc] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000002, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:25.544 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-11-27 13:36:25.589 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,fbb8a14d33be] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000014, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=8000009, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:24.771+08:00, sub_task_instance_id=10000014}, listMap={cluster_ids=[1, 2]}}, executor=10.120.0.1 2023-11-27 13:36:25.599 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5] 2023-11-27 13:36:25.600 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5] 2023-11-27 13:36:25.660 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000002, status:SUCCESS 2023-11-27 13:36:25.662 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000002, status:SUCCESS 2023-11-27 13:36:25.942 INFO 10 --- [pool-scanner--1280-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=464 2023-11-27 13:36:25.952 INFO 10 --- [pool-scanner--1285-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=464 2023-11-27 13:36:25.953 INFO 10 --- [pool-scanner--1283-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=464 2023-11-27 13:36:25.954 INFO 10 --- [pool-scanner--1284-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=464 2023-11-27 13:36:25.978 INFO 10 --- [pool-scanner--1278-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=464 2023-11-27 13:36:26.036 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000010, status:SUCCESS 2023-11-27 13:36:26.056 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000010, status:SUCCESS 2023-11-27 13:36:26.101 INFO 10 --- [pool-scanner--1288-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=464 2023-11-27 13:36:26.103 INFO 10 --- [pool-scanner--1289-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=464 2023-11-27 13:36:26.129 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for LatchSnapshotOperator, error count is 1 2023-11-27 13:36:26.129 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for ParameterSnapshotOperator 2023-11-27 13:36:26.129 WARN 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid time to generate snapshot data, snapshot end interval time is 1701063366973000, now is 1701063386129000 2023-11-27 13:36:26.254 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:26.254 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:26.271 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2023-11-27 13:36:26.332 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:36:07.0, nextRun=2023-11-27 13:36:25.0, currentDate=Mon Nov 27 13:36:26 CST 2023 2023-11-27 13:36:26.332 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-11-27 13:36:07.0, nextRunTime=2023-11-27 13:36:25.0} 2023-11-27 13:36:26.822 INFO 10 --- [http-nio-0.0.0.0-8080-exec-8,1b5692d4175b4fff,af8be6078d24] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.11, traceId=1b5692d4175b4fff, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:36:26.825 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for ParameterSnapshotOperator, error count is 1 2023-11-27 13:36:26.825 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for RowLockSnapshotOperator 2023-11-27 13:36:26.825 WARN 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid time to generate snapshot data, snapshot end interval time is 1701063366973000, now is 1701063386825000 2023-11-27 13:36:26.914 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-11-27 13:36:26.914 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:26.914 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:27.134 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-11-27 13:36:27.247 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2023-11-27 13:36:27.291 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1701187200000000 2023-11-27 13:36:27.311 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000026, status:SUCCESS 2023-11-27 13:36:27.339 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5] 2023-11-27 13:36:27.349 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:27.350 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:27.372 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1701187200000000 values less than (1701187200000000)) 2023-11-27 13:36:27.392 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000026, status:SUCCESS 2023-11-27 13:36:27.495 INFO 10 --- [http-nio-0.0.0.0-8080-exec-8,1b5692d4175b4fff,af8be6078d24] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.11, traceId=1b5692d4175b4fff, duration=674 ms] 2023-11-27 13:36:27.542 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000033, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:27.586 INFO 10 --- [pool-scanner--1291-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:27.597 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000033, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-11-27T13:36:27.506+08:00, endTime=null} 2023-11-27 13:36:27.615 INFO 10 --- [pool-scanner--1292-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:27.620 INFO 10 --- [pool-scanner--1293-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:27.633 INFO 10 --- [pool-scanner--1294-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:27.688 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for RowLockSnapshotOperator, error count is 1 2023-11-27 13:36:27.688 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for ClusterInfoSnapshotOperator 2023-11-27 13:36:27.688 WARN 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid time to generate snapshot data, snapshot end interval time is 1701063366973000, now is 1701063387688000 2023-11-27 13:36:27.934 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000033, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000027, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:27.503+08:00, sub_task_instance_id=10000033}, listMap={host_ids=[1, 2, 3, 4, 5, 6, 7, 9]}}, executor=10.120.0.1 2023-11-27 13:36:27.977 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:27.977 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:27.987 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2023-11-27 13:36:28.134 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.a.i.o.ClusterInfoOperator : ob_hist_cluster_snapshot generate data success, clusterName = obcluster, snapshotId = 8140, endIntervalTime = 1701063366973000 2023-11-27 13:36:28.134 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for ClusterInfoSnapshotOperator, error count is 1 2023-11-27 13:36:28.134 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for TenantInfoSnapshotOperator 2023-11-27 13:36:28.134 WARN 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid time to generate snapshot data, snapshot end interval time is 1701063366973000, now is 1701063388134000 2023-11-27 13:36:28.209 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:36:28.275 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:36:28.519 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2023-11-27 13:36:28.650 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1701273600000000 2023-11-27 13:36:28.666 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:28.666 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:28.669 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1701273600000000 values less than (1701273600000000)) 2023-11-27 13:36:28.782 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:28.785 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:28.796 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:28.797 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:28.807 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:28.865 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:28.866 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:28.884 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270813, maxInactiveSeconds=180 2023-11-27 13:36:28.884 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270813, maxInactiveSeconds=180 2023-11-27 13:36:28.884 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270813, maxInactiveSeconds=180 2023-11-27 13:36:28.884 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270813, maxInactiveSeconds=180 2023-11-27 13:36:28.884 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270813, maxInactiveSeconds=180 2023-11-27 13:36:28.884 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270813, maxInactiveSeconds=180 2023-11-27 13:36:28.884 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270813, maxInactiveSeconds=180 2023-11-27 13:36:28.884 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270813, maxInactiveSeconds=180 2023-11-27 13:36:28.884 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270453, maxInactiveSeconds=180 2023-11-27 13:36:28.884 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270453, maxInactiveSeconds=180 2023-11-27 13:36:28.884 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270453, maxInactiveSeconds=180 2023-11-27 13:36:28.884 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270453, maxInactiveSeconds=180 2023-11-27 13:36:28.884 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270453, maxInactiveSeconds=180 2023-11-27 13:36:28.884 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270453, maxInactiveSeconds=180 2023-11-27 13:36:28.884 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270453, maxInactiveSeconds=180 2023-11-27 13:36:28.884 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270453, maxInactiveSeconds=180 2023-11-27 13:36:28.884 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270453, maxInactiveSeconds=180 2023-11-27 13:36:28.884 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:56, inactiveSeconds=270452, maxInactiveSeconds=180 2023-11-27 13:36:28.885 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:56, inactiveSeconds=270452, maxInactiveSeconds=180 2023-11-27 13:36:28.885 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270453, maxInactiveSeconds=180 2023-11-27 13:36:28.885 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270453, maxInactiveSeconds=180 2023-11-27 13:36:28.885 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270453, maxInactiveSeconds=180 2023-11-27 13:36:28.885 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270453, maxInactiveSeconds=180 2023-11-27 13:36:28.885 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270453, maxInactiveSeconds=180 2023-11-27 13:36:28.885 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270453, maxInactiveSeconds=180 2023-11-27 13:36:28.885 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270453, maxInactiveSeconds=180 2023-11-27 13:36:28.885 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270453, maxInactiveSeconds=180 2023-11-27 13:36:28.885 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270453, maxInactiveSeconds=180 2023-11-27 13:36:28.885 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270453, maxInactiveSeconds=180 2023-11-27 13:36:28.885 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270453, maxInactiveSeconds=180 2023-11-27 13:36:28.885 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270453, maxInactiveSeconds=180 2023-11-27 13:36:28.885 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270453, maxInactiveSeconds=180 2023-11-27 13:36:28.885 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270453, maxInactiveSeconds=180 2023-11-27 13:36:28.885 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270453, maxInactiveSeconds=180 2023-11-27 13:36:28.885 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270453, maxInactiveSeconds=180 2023-11-27 13:36:28.885 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270453, maxInactiveSeconds=180 2023-11-27 13:36:28.885 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270453, maxInactiveSeconds=180 2023-11-27 13:36:28.885 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270453, maxInactiveSeconds=180 2023-11-27 13:36:28.885 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270453, maxInactiveSeconds=180 2023-11-27 13:36:28.885 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:28:55, inactiveSeconds=270453, maxInactiveSeconds=180 2023-11-27 13:36:28.885 WARN 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Host [id=1, ip=133.197.206.97, status=OFFLINE] not online, skip check exporter 2023-11-27 13:36:28.885 WARN 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Host [id=2, ip=133.197.204.3, status=OFFLINE] not online, skip check exporter 2023-11-27 13:36:28.885 WARN 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Host [id=3, ip=133.197.206.92, status=OFFLINE] not online, skip check exporter 2023-11-27 13:36:28.889 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=133.197.204.10] desired exporter count: 8, actual exporter count: 0 2023-11-27 13:36:28.889 WARN 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Send exporter not enough alarm, ip=133.197.204.10, deployedServices=[OB_CLUSTER, OB_PROXY], expectedCount=8, activeCount=0 2023-11-27 13:36:28.894 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+read_consistency(weak) */ unit_id, svr_ip, svr_port, zone, tenant_id, max_cpu, min_cpu, max_memory, min_memory FROM oceanbase.`gv$unit` 2023-11-27 13:36:29.274 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=133.197.204.11] desired exporter count: 8, actual exporter count: 0 2023-11-27 13:36:29.274 WARN 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Send exporter not enough alarm, ip=133.197.204.11, deployedServices=[OB_CLUSTER, OB_PROXY], expectedCount=8, activeCount=0 2023-11-27 13:36:29.437 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,1bf10434ee48] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000020, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:29.469 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:29.469 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:29.481 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2023-11-27 13:36:29.540 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=6, ip=133.197.204.12] desired exporter count: 8, actual exporter count: 0 2023-11-27 13:36:29.541 WARN 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Send exporter not enough alarm, ip=133.197.204.12, deployedServices=[OB_CLUSTER, OB_PROXY], expectedCount=8, activeCount=0 2023-11-27 13:36:29.764 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2023-11-27 13:36:29.779 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1698163200000000 2023-11-27 13:36:29.781 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:29.782 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:29.787 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` drop partition (p1698163200000000) 2023-11-27 13:36:29.789 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=7, ip=133.197.204.13] desired exporter count: 8, actual exporter count: 0 2023-11-27 13:36:29.789 WARN 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Send exporter not enough alarm, ip=133.197.204.13, deployedServices=[OB_CLUSTER, OB_PROXY], expectedCount=8, activeCount=0 2023-11-27 13:36:30.755 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063390, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:30.757 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063390, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:30.757 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063390, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:30.763 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063390, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:30.763 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063390, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:31.079 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] slow query, durationMillis=1293, sql=alter table `ob_cluster_system_event` drop partition (p1698163200000000) 2023-11-27 13:36:31.084 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:31.085 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:31.094 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2023-11-27 13:36:31.161 WARN 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.s.t.b.host.CheckExporterTask : Host [id=9, ip=133.197.206.88, status=OFFLINE] not online, skip check exporter 2023-11-27 13:36:31.185 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,064fdc5b0f20] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000033, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:31.208 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000014, status:SUCCESS 2023-11-27 13:36:31.210 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000014, status:SUCCESS 2023-11-27 13:36:31.275 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:36:31.287 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:36:31.298 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000027, status:SUCCESS 2023-11-27 13:36:31.300 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000027, status:SUCCESS 2023-11-27 13:36:31.512 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1698249600000000 2023-11-27 13:36:31.514 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:31.515 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:31.516 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` drop partition (p1698249600000000) 2023-11-27 13:36:31.924 INFO 10 --- [pool-overview-container-2,,ae18ce4e4b19] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=4 2023-11-27 13:36:31.924 INFO 10 --- [pool-overview-container-2,,ae18ce4e4b19] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:36:32.327 INFO 10 --- [pool-scanner--1279-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=9572 2023-11-27 13:36:32.341 INFO 10 --- [pool-scanner--1290-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=9572 2023-11-27 13:36:32.648 INFO 10 --- [http-nio-0.0.0.0-8080-exec-6,f1d9aac5ae734554,f95c5b13c15c] c.o.o.server.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=darenOB&version=2&ObClusterId=10, client=133.197.204.10, traceId=f1d9aac5ae734554, method=OcpApiRtnObj com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=darenOB,10,2,{"ObRegion":"darenOB","ObCluster":"darenOB","ObRegionId":10,"ObClusterId":10,"Type":"PRIMARY","timestamp":1701063389824169,"RsList":[{"address":"133.197.204.10:2882","role":"LEADER","sql_port":2881},{"address":"133.197.204.11:2882","role":"FOLLOWER","sql_port":2881},{"address":"133.197.204.12:2882","role":"FOLLOWER","sql_port":2881},{"address":"133.197.204.13:2882","role":"FOLLOWER","sql_port":2881}],"ReadonlyRsList":[]},] 2023-11-27 13:36:32.837 INFO 10 --- [http-nio-0.0.0.0-8080-exec-6,f1d9aac5ae734554,f95c5b13c15c] c.o.o.obops.configurl.ConfigUrlService : updateRootServer, clusterName=darenOB, clusterId=10, affectRows=1 2023-11-27 13:36:32.894 INFO 10 --- [pool-scanner--1281-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=9572 2023-11-27 13:36:32.954 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:36:08.0, nextRun=2023-11-27 13:36:30.0, currentDate=Mon Nov 27 13:36:32 CST 2023 2023-11-27 13:36:32.954 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-11-27 13:36:08.0, nextRunTime=2023-11-27 13:36:30.0} 2023-11-27 13:36:33.140 INFO 10 --- [http-nio-0.0.0.0-8080-exec-9,89626eebf06840e9,b8774f91d760] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.13, traceId=89626eebf06840e9, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:36:33.211 INFO 10 --- [pool-ocp-schedules-20,,9c6f6ae4eb75] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:36:33.385 INFO 10 --- [pool-scanner--1282-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=9572 2023-11-27 13:36:33.424 INFO 10 --- [http-nio-0.0.0.0-8080-exec-6,f1d9aac5ae734554,f95c5b13c15c] c.o.o.server.trace.RequestTracingAspect : API OK: [POST /services client=133.197.204.10, traceId=f1d9aac5ae734554, duration=2735 ms] 2023-11-27 13:36:33.541 INFO 10 --- [pool-scanner--1286-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=9572 2023-11-27 13:36:33.608 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-11-27 13:36:33.608 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:33.608 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:33.608 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:33.625 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.a.i.operator.TenantInfoOperator : ob_hist_tenant snapshot generate data success, clusterName = obcluster, snapshotId = 8140, endIntervalTime = 1701063366973000 2023-11-27 13:36:33.625 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for TenantInfoSnapshotOperator, error count is 1 2023-11-27 13:36:33.625 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for ReplicaInfoOperator 2023-11-27 13:36:33.625 WARN 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid time to generate snapshot data, snapshot end interval time is 1701063366973000, now is 1701063393625000 2023-11-27 13:36:33.643 INFO 10 --- [http-nio-0.0.0.0-8080-exec-9,89626eebf06840e9,b8774f91d760] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.13, traceId=89626eebf06840e9, duration=504 ms] 2023-11-27 13:36:33.657 INFO 10 --- [pool-scanner--1295-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=132 2023-11-27 13:36:33.684 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:36:33.722 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:36:33.741 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:33.752 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:33.753 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:33.753 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:33.753 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:33.762 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:33.762 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:33.769 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ READ_CONSISTENCY(WEAK) QUERY_TIMEOUT(60000000) */ p.tenant_id AS ob_tenant_id, p.svr_ip as server_ip, COALESCE(p.replica_count, 0) AS total_replica_count, COALESCE(p.leader_count, 0) AS leader_replica_count, COALESCE(p.follower_count, 0) AS follower_replica_count, COALESCE(p.full_count, 0) AS full_replica_count, COALESCE(p.logonly_count, 0) AS logonly_replica_count, COALESCE(p.readonly_count, 0) AS readonly_replica_count, COALESCE(p.data_size, 0) AS data_size, COALESCE(p.required_size, 0) AS required_size FROM (SELECT tenant_id, svr_ip, COUNT(*) AS replica_count, SUM(case when role = 1 or role = 3 then 1 else 0 end) AS leader_count, SUM(case when role = 1 or role = 3 then 0 else 1 end) AS follower_count, SUM(case when replica_type = 0 then 1 else 0 end) AS full_count, SUM(case when replica_type = 5 then 1 else 0 end) AS logonly_count, SUM(case when replica_type = 16 then 1 else 0 end) AS readonly_count, SUM(data_size) AS data_size, SUM(required_size) AS required_size FROM oceanbase.__all_virtual_tenant_partition_meta_table GROUP BY tenant_id, svr_ip) p ORDER BY ob_tenant_id, svr_ip ASC 2023-11-27 13:36:33.786 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,fbb8a14d33be] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:33.788 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,878f2d940ab2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:33.790 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,fbb8a14d33be] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:33.790 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,fbb8a14d33be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:33.792 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,878f2d940ab2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:33.792 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,878f2d940ab2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:33.794 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,878f2d940ab2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:33.795 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:36:10.0, nextRun=2023-11-27 13:36:30.0, currentDate=Mon Nov 27 13:36:33 CST 2023 2023-11-27 13:36:33.795 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-11-27 13:36:10.0, nextRunTime=2023-11-27 13:36:30.0} 2023-11-27 13:36:33.799 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,fbb8a14d33be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:33.806 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,fbb8a14d33be] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:33.806 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,fbb8a14d33be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:33.807 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,878f2d940ab2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:33.807 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,878f2d940ab2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:33.809 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,fbb8a14d33be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND timestamp > :startTime AND timestamp <= :endTime ORDER BY timestamp ASC LIMIT :limit, args: [2023-11-24 10:16:38.567361, 2023-11-27 13:36:32.779, 1000] 2023-11-27 13:36:33.810 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,878f2d940ab2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ * FROM oceanbase.__all_rootservice_event_history WHERE 1=1 AND not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND gmt_create > :startTime AND gmt_create <= :endTime ORDER BY gmt_create ASC LIMIT :limit, args: [2023-11-24 10:32:32.194483, 2023-11-27 13:36:33.533, 1000] 2023-11-27 13:36:33.974 INFO 10 --- [pool-prometheus-cache-monitor1,,488db88f6e88] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":0,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":87087,"secondWriteQueue":0,"minuteWriteCacheSize":211} 2023-11-27 13:36:33.974 INFO 10 --- [pool-prometheus-cache-monitor1,,488db88f6e88] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_zone_current_timestamp=38, ob_server_resource_memory_assigned_bytes=21, tcp_retrans=30, ob_unit_num=21, go_memstats_mspan_sys_bytes=17, go_memstats_stack_sys_bytes=17, exporter_export_metrics_bytes_total=28, node_filesystem_size_bytes=87, http_server_requests_seconds_count=40, ob_compaction_frozen_time=18, ob_system_event_total_waits=34, monagent_input_sql_duration_seconds_sum=60, observer_fd_count=20, ob_plan_cache_hit_total=28, processor_process_metrics_total=22, node_filesystem_readonly=86, ob_server_resource_disk_in_use_bytes=19, go_memstats_frees_total=17, sql_audit_input_collect_data_jumped_total=12, io_await=60, host_available=13, go_memstats_mallocs_total=17, mysql_output_write_sql_failed_count=23, ob_unit_config_max_memory_bytes=45, io_util=60, ob_system_event_detail_total_timeouts=782, ob_tenant_disk_v4_data_size=18, ob_latch_sleep_total=326, ob_tenant_unit_min_memory=41, ob_memstore_total_bytes=28, process_resident_memory_bytes=17, ob_cache_size_bytes=47, ob_tenant_unit_max_memory=41, ob_tenant_assigned_cpu_total=25, input_collect_metrics_total=29, ob_tenant_server_available_disk_free=28, ob_tenant_server_avg_cpu_used_percentage=23, node_disk_io_time_weighted_seconds_total=39, node_disk_write_time_seconds_total=39, cpu_count=30, output_write_metrics_total=15, ob_cluster_sync=175, process_open_fds=17, odp_transaction_total=20, ob_system_event_detail_time_waited_us=782, process_max_fds=17, ob_server_resource_cpu_assigned_percent=20, ocp_alarm_detect_duration_ms_count=82, go_memstats_heap_inuse_bytes=17, ob_tenant_server_max_cpu_used_percentage=23, proxy_memory_used_bytes=30, go_memstats_mcache_sys_bytes=17, ob_tenant_disk_v4_log_size=18, node_memory_memfree_bytes=29, process_exists=38, go_gc_duration_seconds=22, ob_server_resource_disk_bytes=21, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=27, node_memory_cached_bytes=29, ob_table_num=24, ob_disk_free_bytes=21, node_filesystem_avail_bytes=87, ob_backup_storage_capacity_bytes=349, ob_active_session_num=27, ob_unit_config_max_iops=45, ob_plan_cache_memory_bytes=28, node_network_transmit_errs_total=65, go_memstats_last_gc_time_seconds=17, go_gc_duration_seconds_count=17, monitor_exporter_avaliable=78, ob_tenant_server_max_memory_used_percentage=23, node_disk_read_bytes_total=39, mysql_output_write_sql_milliseconds_summary_count=25, ob_compaction_is_suspended=18, ob_tenant_unit_log_disk_size=22, go_memstats_lookups_total=17, mysql_output_write_sql_count=25, log_tailer_count=17, host_agent_process_status=21, ob_server_resource_cpu=21, go_memstats_alloc_bytes=17, ob_server_num=58, node_cpu_seconds_total_native=47, ob_tenant_unit_count=39, ob_session_all_num=25, ob_tenant_resource_min_memory=25, ob_memstore_freeze_times=28, ob_system_event_time_waited=34, ob_tenant_resource_min_cpu=25, ob_cluster_status_check=49, monagent_plugin_execute_seconds_total=41, node_net_bandwidth_bps=45, monagent_oceanbase_db_pool_idle=7, go_memstats_heap_idle_bytes=17, go_memstats_gc_sys_bytes=17, ob_server_resource_memory_bytes=21, ob_tenant_sys_partition_required_size=12, ob_latch_get_total=326, go_info=18, ob_tenant_assigned_mem_assigned=25, log_tailer_processed_file_count=24, node_load15=29, ob_tenant_server_required_size=35, ob_unit_config_min_iops=45, ob_server_partition_replica_limit=7, ob_tenant_unit_max_cpu=41, ob_latch_wait_seconds_total=326, node_memory_buffers_bytes=29, go_memstats_heap_released_bytes=17, node_network_receive_bytes_total=65, ob_memstore_freeze_trigger_bytes=28, ob_waitevent_wait_seconds_total=28, ob_server_stopped_duration_seconds=20, exposer_request_latencies_count=8, ob_role_is_rootserver=20, go_memstats_next_gc_bytes=17, mysql_output_context_timeout_discard_metrics_count=17, ob_session_active_num=25, node_filesystem_files_free=109, ob_memtable_max_snapshot_duration_seconds=27, node_disk_read_time_seconds_total=39, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=22, process_virtual_memory_bytes=17, mysql_output_write_sql_retry_count=23, ob_tenant_assigned_cpu_assigned=25, ob_tenant_unit_min_cpu=41, ob_unit_config_max_cpu=45, ob_unit_config_max_session_num=18, http_request_milliseconds_summary_count=79, monagent_oceanbase_audit_timeinfo_cache_size=7, net_connectivity=10, monagent_input_sql_duration_seconds_bucket=70, ob_latch_immediate_get_total=326, node_cpu_seconds_total=37, http_request_milliseconds_summary=82, ob_latch_immediate_miss_total=326, ob_index_error_num=16, ob_tenant_server_available_disk_total=28, ob_latch_spin_get_total=326, ob_tenant_memory_hold_bytes=18, go_memstats_buck_hash_sys_bytes=17, monagent_pipeline_report_metrics_total=18, mysql_output_write_sql_milliseconds_summary_sum=25, ob_tenant_disk_total_size=25, exposer_transferred_bytes_total=8, odp_sql_request_total=67, mysql_output_write_task_size=24, ob_tenant_context_memory_count=339, node_disk_reads_completed_total=39, exporter_export_metrics_total=28, ob_sysstat=129, ob_partition_num=13, node_ntp_offset_seconds=29, ob_latch_miss_total=326, go_memstats_heap_sys_bytes=17, ocp_meta_db_health=4, ob_disk_total_bytes=21, ob_zone_stat=38, go_memstats_heap_objects=17, go_memstats_mcache_inuse_bytes=17, go_memstats_other_sys_bytes=17, go_threads=17, http_request_milliseconds_summary_sum=79, ob_memstore_active_bytes=28, http_server_requests_seconds_sum=40, go_memstats_alloc_bytes_total=17, ob_tenant_server_partition_replica_count=35, ob_tenant_assigned_mem_total=25, node_memory_memtotal_bytes=29, node_network_transmit_drop_total=65, ob_system_event_detail_total_waits=782, odp_used_connections=28, ob_tenant_task_max_duration_seconds=25, backup_storage_capacity_used_percent=14, exposer_request_latencies=11, ob_waitevent_wait_total=28, odp_sql_cost_total=66, ob_tenant500_memory_used_bytes=20, node_network_transmit_bytes_total=65, ob_compaction_is_error=18, node_network_receive_drop_total=65, ob_server_partition_replica_count=7, monagent_pipeline_execute_total=27, node_filesystem_files=109, ob_tenant_server_avg_memory_used_percentage=23, process_start_time_seconds=17, go_memstats_heap_alloc_bytes=17, ob_tenant_resource_max_cpu=25, go_gc_duration_seconds_sum=17, go_memstats_mspan_inuse_bytes=17, mysql_output_write_sql_milliseconds_summary=28, ob_server_resource_memory_assigned_percent=20, monagent_pipeline_execute_seconds_total=27, go_memstats_sys_bytes=17, node_network_receive_errs_total=65, ob_all_session_num=27, observer_ulimit_max_fd_count=20, ob_plan_cache_access_total=28, ob_tenant_server_row_count=35, ob_tenant_memory_used_bytes=18, monagent_oceanbase_plan_cache_size=7, ob_tenant_server_data_size=35, go_memstats_gc_cpu_fraction=17, monagent_oceanbase_audit_cache_size=7, ob_tenant_context_memory_used_bytes=339, node_disk_writes_completed_total=39, node_load1=29, process_cpu_seconds_total=17, ob_tenant_resource_max_memory=25, go_memstats_stack_inuse_bytes=17, ob_tenant_user_partition_required_size=12, node_load5=29, monagent_input_sql_duration_seconds_count=60, node_disk_written_bytes_total=39, monagent_oceanbase_ash_cache_size=7, ob_tenant_context_memory_hold_bytes=339, go_goroutines=17, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=21, exposer_request_latencies_sum=8, ob_unit_config_min_cpu=45, process_virtual_memory_max_bytes=17, monagent_oceanbase_plan_timeinfo_cache_size=7, monagent_plugin_execute_total=41, ob_tenant500_memory_hold_bytes=20, exposer_scrapes_total=8, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=22, ob_compaction_start_time=18, oceanbase_connectivity=21} 2023-11-27 13:36:34.019 INFO 10 --- [pool-scanner--1277-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=9572 2023-11-27 13:36:34.041 INFO 10 --- [pool-scanner--1287-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=9572 2023-11-27 13:36:34.044 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-11-27 13:36:34.044 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:36:34.044 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-11-27 13:36:34.067 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for LatchSnapshotOperator, error count is 0 2023-11-27 13:36:34.067 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for ParameterSnapshotOperator 2023-11-27 13:36:34.067 WARN 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid time to generate snapshot data, snapshot end interval time is 1701063366973000, now is 1701063394067000 2023-11-27 13:36:34.238 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:36:34.289 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:36:34.290 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:36:34.415 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:36:34.550 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:36:34.549027+08:00, lastProcessTime=2023-11-27T13:36:23.250440+08:00 2023-11-27 13:36:34.742 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:36:34.877 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for ParameterSnapshotOperator, error count is 0 2023-11-27 13:36:34.877 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for RowLockSnapshotOperator 2023-11-27 13:36:34.877 WARN 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid time to generate snapshot data, snapshot end interval time is 1701063366973000, now is 1701063394877000 2023-11-27 13:36:35.103 INFO 10 --- [pool-prometheus-second-schedule-2,,2f16bb323edd] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=3, collectTaskCount=3 2023-11-27 13:36:35.104 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063395, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:35.104 INFO 10 --- [pool-prometheus-second-schedule-2,,20c8f5649287] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=2, collectTaskCount=2 2023-11-27 13:36:35.104 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063395, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:35.106 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063395, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:35.106 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063395, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:35.109 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063395, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:35.382 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000034, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:35.464 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-11-27 13:36:35.484 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:36:35.489 INFO 10 --- [pool-ocp-schedules-14,,5200c603e7c6] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:36:35.490 INFO 10 --- [pool-ocp-schedules-14,,5200c603e7c6] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:36:35.506 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000034, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-11-27T13:36:35.361+08:00, endTime=null} 2023-11-27 13:36:35.577 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000035, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:35.589 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5] 2023-11-27 13:36:35.937 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:36:34.549027+08:00 2023-11-27 13:36:36.098 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000035, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-11-27T13:36:35.573+08:00, endTime=null} 2023-11-27 13:36:36.256 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] slow query, durationMillis=4740, sql=alter table `ob_cluster_system_event` drop partition (p1698249600000000) 2023-11-27 13:36:36.257 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:36.258 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:36.260 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2023-11-27 13:36:36.280 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000034, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=8000028, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:35.360+08:00, sub_task_instance_id=10000034}, listMap={cluster_ids=[1, 2]}}, executor=10.120.0.1 2023-11-27 13:36:36.280 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000036, operation:EXECUTE, state: RUNNING 2023-11-27 13:36:36.292 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000036, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-11-27T13:36:36.246+08:00, endTime=null} 2023-11-27 13:36:36.339 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000035, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=8000028, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:35.573+08:00, sub_task_instance_id=10000035}, listMap={cluster_ids=[1, 2]}}, executor=10.120.0.1 2023-11-27 13:36:36.466 INFO 10 --- [pool-scanner--1355-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=24 2023-11-27 13:36:36.467 INFO 10 --- [pool-scanner--1354-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:36.479 INFO 10 --- [pool-scanner--1353-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:36.494 INFO 10 --- [pool-scanner--1348-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:36.562 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1698336000000000 2023-11-27 13:36:36.575 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:36.575 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:36.585 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` drop partition (p1698336000000000) 2023-11-27 13:36:36.592 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for RowLockSnapshotOperator, error count is 0 2023-11-27 13:36:36.592 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for ClusterInfoSnapshotOperator 2023-11-27 13:36:36.592 WARN 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid time to generate snapshot data, snapshot end interval time is 1701063366973000, now is 1701063396592000 2023-11-27 13:36:36.604 INFO 10 --- [pool-subtask-executor28,554481c3fd1b426b,40da8e0fc3e0] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000036, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000029, task_operation=execute, latest_execution_start_time=2023-11-27T13:36:36.227+08:00, sub_task_instance_id=10000036}, listMap={}}, executor=10.120.0.1 2023-11-27 13:36:36.604 INFO 10 --- [pool-subtask-executor28,554481c3fd1b426b,40da8e0fc3e0] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-11-27 13:36:36.604 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.a.i.o.ClusterInfoOperator : ob_hist_cluster_snapshot generate data success, clusterName = darenOB, snapshotId = 8139, endIntervalTime = 1701063366973000 2023-11-27 13:36:36.604 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for ClusterInfoSnapshotOperator, error count is 0 2023-11-27 13:36:36.604 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for TenantInfoSnapshotOperator 2023-11-27 13:36:36.604 WARN 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid time to generate snapshot data, snapshot end interval time is 1701063366973000, now is 1701063396604000 2023-11-27 13:36:36.675 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:36.676 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:36.676 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:36.676 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:36.678 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:36.679 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:36.680 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, tenant_name, (CASE WHEN tenant_id = 1 THEN 'SYS' ELSE 'USER' END) AS tenant_type, (CASE WHEN compatibility_mode = 1 THEN 'ORACLE' ELSE 'MYSQL' END) AS compatibility_mode, zone_list, primary_zone, locality, status, locked, read_only, in_recyclebin, TIMESTAMPDIFF(SECOND, gmt_create, now()) AS exist_seconds FROM oceanbase.__all_tenant 2023-11-27 13:36:36.685 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:36.685 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:36.687 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1] 2023-11-27 13:36:36.709 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:36.709 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:36.709 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2023-11-27 13:36:36.709 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:36.710 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:36.715 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1] 2023-11-27 13:36:36.723 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] slow query, durationMillis=2954, sql=SELECT /*+ READ_CONSISTENCY(WEAK) QUERY_TIMEOUT(60000000) */ p.tenant_id AS ob_tenant_id, p.svr_ip as server_ip, COALESCE(p.replica_count, 0) AS total_replica_count, COALESCE(p.leader_count, 0) AS leader_replica_count, COALESCE(p.follower_count, 0) AS follower_replica_count, COALESCE(p.full_count, 0) AS full_replica_count, COALESCE(p.logonly_count, 0) AS logonly_replica_count, COALESCE(p.readonly_count, 0) AS readonly_replica_count, COALESCE(p.data_size, 0) AS data_size, COALESCE(p.required_size, 0) AS required_size FROM (SELECT tenant_id, svr_ip, COUNT(*) AS replica_count, SUM(case when role = 1 or role = 3 then 1 else 0 end) AS leader_count, SUM(case when role = 1 or role = 3 then 0 else 1 end) AS follower_count, SUM(case when replica_type = 0 then 1 else 0 end) AS full_count, SUM(case when replica_type = 5 then 1 else 0 end) AS logonly_count, SUM(case when replica_type = 16 then 1 else 0 end) AS readonly_count, SUM(data_size) AS data_size, SUM(required_size) AS required_size FROM oceanbase.__all_virtual_tenant_partition_meta_table GROUP BY tenant_id, svr_ip) p ORDER BY ob_tenant_id, svr_ip ASC 2023-11-27 13:36:36.733 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:36.733 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:36.735 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:36.735 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:36.735 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ READ_CONSISTENCY(WEAK) QUERY_TIMEOUT(60000000) */ p.tenant_id AS ob_tenant_id, p.svr_ip as server_ip, COALESCE(p.replica_count, 0) AS total_replica_count, COALESCE(p.leader_count, 0) AS leader_replica_count, COALESCE(p.follower_count, 0) AS follower_replica_count, COALESCE(p.full_count, 0) AS full_replica_count, COALESCE(p.logonly_count, 0) AS logonly_replica_count, COALESCE(p.readonly_count, 0) AS readonly_replica_count, COALESCE(p.data_size, 0) AS data_size, COALESCE(p.required_size, 0) AS required_size FROM (SELECT tenant_id, svr_ip, COUNT(*) AS replica_count, SUM(case when role = 1 or role = 3 then 1 else 0 end) AS leader_count, SUM(case when role = 1 or role = 3 then 0 else 1 end) AS follower_count, SUM(case when replica_type = 0 then 1 else 0 end) AS full_count, SUM(case when replica_type = 5 then 1 else 0 end) AS logonly_count, SUM(case when replica_type = 16 then 1 else 0 end) AS readonly_count, SUM(data_size) AS data_size, SUM(required_size) AS required_size FROM oceanbase.__all_root_table GROUP BY tenant_id, svr_ip) p ORDER BY ob_tenant_id, svr_ip ASC 2023-11-27 13:36:36.735 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:36.735 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:36.744 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:36.750 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:36.750 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:36.752 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1001] 2023-11-27 13:36:36.753 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2023-11-27 13:36:36.757 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:36.757 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:36.765 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ READ_CONSISTENCY(WEAK) QUERY_TIMEOUT(60000000) */ p.tenant_id AS ob_tenant_id, p.svr_ip as server_ip, COALESCE(p.replica_count, 0) AS total_replica_count, COALESCE(p.leader_count, 0) AS leader_replica_count, COALESCE(p.follower_count, 0) AS follower_replica_count, COALESCE(p.full_count, 0) AS full_replica_count, COALESCE(p.logonly_count, 0) AS logonly_replica_count, COALESCE(p.readonly_count, 0) AS readonly_replica_count, COALESCE(p.data_size, 0) AS data_size, COALESCE(p.required_size, 0) AS required_size FROM (SELECT tenant_id, svr_ip, COUNT(*) AS replica_count, SUM(case when role = 1 or role = 3 then 1 else 0 end) AS leader_count, SUM(case when role = 1 or role = 3 then 0 else 1 end) AS follower_count, SUM(case when replica_type = 0 then 1 else 0 end) AS full_count, SUM(case when replica_type = 5 then 1 else 0 end) AS logonly_count, SUM(case when replica_type = 16 then 1 else 0 end) AS readonly_count, SUM(data_size) AS data_size, SUM(required_size) AS required_size FROM oceanbase.__all_virtual_core_meta_table GROUP BY tenant_id, svr_ip) p ORDER BY ob_tenant_id, svr_ip ASC 2023-11-27 13:36:36.791 INFO 10 --- [pool-208-thread-2,554481c3fd1b426b,a0041e22e30d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:darenOB, tenant:daren 2023-11-27 13:36:36.821 INFO 10 --- [pool-208-thread-1,554481c3fd1b426b,164e4194f45d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:darenOB, tenant:sys 2023-11-27 13:36:36.827 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:36.827 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:36.828 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1001] 2023-11-27 13:36:36.854 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:36.854 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:36.861 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1002] 2023-11-27 13:36:36.888 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:36.888 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:36.890 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ READ_CONSISTENCY(WEAK) QUERY_TIMEOUT(60000000) */ p.tenant_id AS ob_tenant_id, p.svr_ip as server_ip, COALESCE(p.replica_count, 0) AS total_replica_count, COALESCE(p.leader_count, 0) AS leader_replica_count, COALESCE(p.follower_count, 0) AS follower_replica_count, COALESCE(p.full_count, 0) AS full_replica_count, COALESCE(p.logonly_count, 0) AS logonly_replica_count, COALESCE(p.readonly_count, 0) AS readonly_replica_count, COALESCE(p.data_size, 0) AS data_size, COALESCE(p.required_size, 0) AS required_size FROM (SELECT tenant_id, svr_ip, COUNT(*) AS replica_count, SUM(case when role = 1 or role = 3 then 1 else 0 end) AS leader_count, SUM(case when role = 1 or role = 3 then 0 else 1 end) AS follower_count, SUM(case when replica_type = 0 then 1 else 0 end) AS full_count, SUM(case when replica_type = 5 then 1 else 0 end) AS logonly_count, SUM(case when replica_type = 16 then 1 else 0 end) AS readonly_count, SUM(data_size) AS data_size, SUM(required_size) AS required_size FROM oceanbase.__all_virtual_core_root_table GROUP BY tenant_id, svr_ip) p ORDER BY ob_tenant_id, svr_ip ASC 2023-11-27 13:36:36.908 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:36.908 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:36.909 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1002] 2023-11-27 13:36:36.921 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-11-27 13:36:36.976 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:36.986 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:36.989 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:36.990 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:36.991 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:36.991 INFO 10 --- [pool-208-thread-1,554481c3fd1b426b,164e4194f45d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:36.994 INFO 10 --- [pool-208-thread-1,554481c3fd1b426b,164e4194f45d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:36.994 INFO 10 --- [pool-208-thread-1,554481c3fd1b426b,164e4194f45d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:36.995 INFO 10 --- [pool-208-thread-1,554481c3fd1b426b,164e4194f45d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:36.996 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:36.996 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:37.001 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT UNIT_ID, SVR_IP, SVR_PORT, ZONE, TENANT_ID, MIN_CPU, MAX_CPU, MIN_MEMORY, MAX_MEMORY, MIN_IOPS, MAX_IOPS, IOPS_WEIGHT, DATA_DISK_IN_USE, LOG_DISK_IN_USE FROM (SELECT T1.UNIT_ID, T1.SVR_IP, T1.SVR_PORT, T1.ZONE, T2.TENANT_ID, T1.MIN_CPU, T1.MAX_CPU, T1.MIN_MEMORY, T1.MAX_MEMORY, T1.MIN_IOPS, T1.MAX_IOPS, T1.IOPS_WEIGHT, T1.DATA_DISK_IN_USE, T1.LOG_DISK_IN_USE FROM (SELECT UNIT_ID, SVR_IP, SVR_PORT, ZONE, SUM(MIN_CPU) AS MIN_CPU, SUM(MAX_CPU) AS MAX_CPU, SUM(MEMORY_SIZE) AS MIN_MEMORY, SUM(MEMORY_SIZE) AS MAX_MEMORY, SUM(MIN_IOPS) AS MIN_IOPS, SUM(MAX_IOPS) AS MAX_IOPS, SUM(IOPS_WEIGHT) AS IOPS_WEIGHT, SUM(DATA_DISK_IN_USE) AS DATA_DISK_IN_USE, SUM(LOG_DISK_IN_USE) AS LOG_DISK_IN_USE FROM oceanbase.GV$OB_UNITS GROUP BY UNIT_ID, SVR_IP, SVR_PORT ) T1 JOIN oceanbase.DBA_OB_UNITS T2 ON T1.UNIT_ID = T2.UNIT_ID AND T1.SVR_IP = T2.SVR_IP AND T1.SVR_PORT = T2.SVR_PORT) 2023-11-27 13:36:37.009 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:37.012 INFO 10 --- [pool-208-thread-1,554481c3fd1b426b,164e4194f45d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:37.013 INFO 10 --- [pool-208-thread-1,554481c3fd1b426b,164e4194f45d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:37.013 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:37.014 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:37.014 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:37.015 INFO 10 --- [pool-208-thread-1,554481c3fd1b426b,164e4194f45d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-11-27 13:36:37.016 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:37.029 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:37.029 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:37.036 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-11-27 13:36:37.085 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.a.i.operator.TenantInfoOperator : ob_hist_tenant snapshot generate data success, clusterName = darenOB, snapshotId = 8139, endIntervalTime = 1701063366973000 2023-11-27 13:36:37.085 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for TenantInfoSnapshotOperator, error count is 0 2023-11-27 13:36:37.085 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for ReplicaInfoOperator 2023-11-27 13:36:37.086 WARN 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid time to generate snapshot data, snapshot end interval time is 1701063366973000, now is 1701063397086000 2023-11-27 13:36:37.127 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:37.128 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:37.129 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:37.129 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:37.131 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:37.134 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:37.134 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:37.135 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:36:37.168 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:37.169 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:37.172 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:37.172 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:37.173 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:37.174 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:37.177 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:37.177 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:37.178 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:37.178 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:37.185 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-11-27 13:36:37.185 INFO 10 --- [pool-208-thread-1,554481c3fd1b426b,164e4194f45d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1701021601748952808, broadcast scn:1701021601748952808, saved frozen scn:1701021601748952808 2023-11-27 13:36:37.185 INFO 10 --- [pool-208-thread-1,554481c3fd1b426b,164e4194f45d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:darenOB, tenant:sys 2023-11-27 13:36:37.193 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:37.193 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:37.195 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-11-27 13:36:39.298 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] slow query, durationMillis=2121, sql=set ob_query_timeout = ? 2023-11-27 13:36:39.299 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:39.299 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, sql = delete /*+ QUERY_TIMEOUT(10000000) */ from task_instance where create_time <= date_sub(curdate(), interval 30 day) and type = "SYS_SCHEDULED" limit 1000;, deletedLineNum = 1000 2023-11-27 13:36:39.304 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.120.0.1:8080 2023-11-27 13:36:39.312 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:36:39.312 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] slow query, durationMillis=2559, sql=show create table ocp_metric_data_1 2023-11-27 13:36:39.313 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_1, partition=p1701100800 2023-11-27 13:36:39.315 INFO 10 --- [http-nio-0.0.0.0-8080-exec-2,01fc6417e7dc42ed,c53a58564c43] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.11, traceId=01fc6417e7dc42ed, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:36:39.330 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:36:39.339 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:39.339 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:39.346 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] slow query, durationMillis=2151, sql=SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ? 2023-11-27 13:36:39.349 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_1` add partition(partition p1701100800 values less than (1701100800)) 2023-11-27 13:36:39.351 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:39.351 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:39.363 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT UNIT_ID, SVR_IP, SVR_PORT, ZONE, TENANT_ID, MIN_CPU, MAX_CPU, MIN_MEMORY, MAX_MEMORY, MIN_IOPS, MAX_IOPS, IOPS_WEIGHT, DATA_DISK_IN_USE, LOG_DISK_IN_USE FROM (SELECT T1.UNIT_ID, T1.SVR_IP, T1.SVR_PORT, T1.ZONE, T2.TENANT_ID, T1.MIN_CPU, T1.MAX_CPU, T1.MIN_MEMORY, T1.MAX_MEMORY, T1.MIN_IOPS, T1.MAX_IOPS, T1.IOPS_WEIGHT, T1.DATA_DISK_IN_USE, T1.LOG_DISK_IN_USE FROM (SELECT UNIT_ID, SVR_IP, SVR_PORT, ZONE, SUM(MIN_CPU) AS MIN_CPU, SUM(MAX_CPU) AS MAX_CPU, SUM(MEMORY_SIZE) AS MIN_MEMORY, SUM(MEMORY_SIZE) AS MAX_MEMORY, SUM(MIN_IOPS) AS MIN_IOPS, SUM(MAX_IOPS) AS MAX_IOPS, SUM(IOPS_WEIGHT) AS IOPS_WEIGHT, SUM(DATA_DISK_IN_USE) AS DATA_DISK_IN_USE, SUM(LOG_DISK_IN_USE) AS LOG_DISK_IN_USE FROM oceanbase.GV$OB_UNITS GROUP BY UNIT_ID, SVR_IP, SVR_PORT ) T1 JOIN oceanbase.DBA_OB_UNITS T2 ON T1.UNIT_ID = T2.UNIT_ID AND T1.SVR_IP = T2.SVR_IP AND T1.SVR_PORT = T2.SVR_PORT) 2023-11-27 13:36:39.363 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:39.363 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:39.368 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-11-27 13:36:39.370 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,878f2d940ab2] .o.o.o.i.c.ClusterSystemEventServiceImpl : Batch saved 46 system events to database for cluster 1 2023-11-27 13:36:39.370 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,878f2d940ab2] .o.o.o.i.c.ClusterSystemEventServiceImpl : Sync-ed 46 system events for cluster 1, startTime 2023-11-24 10:32:32.194483, endTime 2023-11-27 13:36:33.533, limit 1000, offset 0 2023-11-27 13:36:39.394 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:39.394 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:39.401 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for ReplicaInfoOperator, error count is 1 2023-11-27 13:36:39.401 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for SysStatSnapshotOperator 2023-11-27 13:36:39.401 WARN 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid time to generate snapshot data, snapshot end interval time is 1701063366973000, now is 1701063399401000 2023-11-27 13:36:39.403 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-11-27 13:36:39.449 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:39.449 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:39.455 INFO 10 --- [pool-208-thread-2,554481c3fd1b426b,a0041e22e30d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:39.457 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-11-27 13:36:39.459 INFO 10 --- [pool-208-thread-2,554481c3fd1b426b,a0041e22e30d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:39.459 INFO 10 --- [pool-208-thread-2,554481c3fd1b426b,a0041e22e30d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:39.464 INFO 10 --- [pool-208-thread-2,554481c3fd1b426b,a0041e22e30d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:39.477 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-11-27 13:36:39.481 INFO 10 --- [pool-208-thread-2,554481c3fd1b426b,a0041e22e30d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:39.481 INFO 10 --- [pool-208-thread-2,554481c3fd1b426b,a0041e22e30d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:39.484 INFO 10 --- [pool-208-thread-2,554481c3fd1b426b,a0041e22e30d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-11-27 13:36:39.526 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:39.532 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:39.544 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:39.544 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:39.551 INFO 10 --- [http-nio-0.0.0.0-8080-exec-2,01fc6417e7dc42ed,c53a58564c43] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.11, traceId=01fc6417e7dc42ed, duration=237 ms] 2023-11-27 13:36:39.561 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-11-27 13:36:39.563 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:39.582 INFO 10 --- [pool-208-thread-2,554481c3fd1b426b,a0041e22e30d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1701021600396785720, broadcast scn:1701021600396785720, saved frozen scn:1701021600396785720 2023-11-27 13:36:39.582 INFO 10 --- [pool-208-thread-2,554481c3fd1b426b,a0041e22e30d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:darenOB, tenant:daren 2023-11-27 13:36:39.587 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:39.588 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:39.596 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1001, ALTER_TENANT_LOCALITY] 2023-11-27 13:36:39.607 INFO 10 --- [pool-subtask-executor28,554481c3fd1b426b,40da8e0fc3e0] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-11-27 13:36:39.613 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,878f2d940ab2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:39.614 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,878f2d940ab2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:39.618 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,878f2d940ab2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:39.618 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,878f2d940ab2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:39.624 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,878f2d940ab2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:39.644 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,878f2d940ab2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:39.644 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,878f2d940ab2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:39.647 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,878f2d940ab2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ * FROM oceanbase.__all_rootservice_event_history WHERE 1=1 AND not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND gmt_create > :startTime AND gmt_create <= :endTime ORDER BY gmt_create ASC LIMIT :limit OFFSET :offset, args: [2023-11-24 10:32:32.194483, 2023-11-27 13:36:33.533, 1000, 1000] 2023-11-27 13:36:39.651 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,878f2d940ab2] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1, startTime 2023-11-24 10:32:32.194483, endTime 2023-11-27 13:36:33.533 2023-11-27 13:36:39.713 INFO 10 --- [pool-subtask-executor28,554481c3fd1b426b,40da8e0fc3e0] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000036, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:39.717 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for ReplicaInfoOperator, error count is 0 2023-11-27 13:36:39.718 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for SysStatSnapshotOperator 2023-11-27 13:36:39.718 WARN 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid time to generate snapshot data, snapshot end interval time is 1701063366973000, now is 1701063399718000 2023-11-27 13:36:39.721 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5] 2023-11-27 13:36:39.727 INFO 10 --- [pool-subtask-executor30,554481c3fd1b426b,878f2d940ab2] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000013, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:39.735 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/stat, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:36:39.735 WARN 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/stat is inactive 2023-11-27 13:36:39.738 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62888/metrics/stat, message=java.net.ConnectException: Connection refused: /133.197.204.3:62888 2023-11-27 13:36:39.738 WARN 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62888/metrics/stat is inactive 2023-11-27 13:36:39.742 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/node/host, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:36:39.742 WARN 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/node/host is inactive 2023-11-27 13:36:39.746 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/obproxy, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:36:39.746 WARN 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/obproxy is inactive 2023-11-27 13:36:39.754 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/node/obproxy, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:36:39.754 WARN 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/node/obproxy is inactive 2023-11-27 13:36:39.818 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-11-27 13:36:39.910 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:39.913 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:39.914 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5] 2023-11-27 13:36:39.915 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:39.915 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:39.918 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:39.923 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:39.924 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:39.926 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1001, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:36:39.961 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,fbb8a14d33be] .o.o.o.i.c.ClusterSystemEventServiceImpl : Batch saved 194 system events to database for cluster 2 2023-11-27 13:36:39.961 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,fbb8a14d33be] .o.o.o.i.c.ClusterSystemEventServiceImpl : Sync-ed 194 system events for cluster 2, startTime 2023-11-24 10:16:38.567361, endTime 2023-11-27 13:36:32.779, limit 1000, offset 0 2023-11-27 13:36:39.981 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:39.992 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:39.995 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:39.995 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:39.996 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:40.000 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:40.000 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:40.002 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-11-27 13:36:40.005 INFO 10 --- [pool-scanner--1392-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:40.052 WARN 10 --- [pool-14-thread-1,,] o.p.s.d.i.OcpPerfSqlDiagnosisTaskService : OcpPerfSqlDiagnosisTaskEntity(taskName=SQL_INDEX_DIAGNOSER, beginIntervalTime=1700793792203000, endIntervalTime=1700794093475000, executor=10.120.0.1, status=SUCCESS, createTime=2023-11-24T02:49:13.478Z, updateTime=2023-11-24T02:49:13.803Z) is too old, skip to diag OcpPerfSqlDiagnosisTaskEntity(taskName=SQL_INDEX_DIAGNOSER, beginIntervalTime=1701061540052000, endIntervalTime=1701063340052000, executor=10.120.0.1, status=RUNNING, createTime=null, updateTime=null) 2023-11-27 13:36:40.053 INFO 10 --- [pool-scanner--1394-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=24 2023-11-27 13:36:40.064 WARN 10 --- [pool-13-thread-1,,] o.p.s.d.i.OcpPerfSqlDiagnosisTaskService : OcpPerfSqlDiagnosisTaskEntity(taskName=PLAN_CHANGE_DETECTOR, beginIntervalTime=1700793855669000, endIntervalTime=1700793976552000, executor=10.120.0.1, status=SUCCESS, createTime=2023-11-24T02:48:16.552Z, updateTime=2023-11-24T02:48:16.884Z) is too old, skip to diag OcpPerfSqlDiagnosisTaskEntity(taskName=PLAN_CHANGE_DETECTOR, beginIntervalTime=1701062680064000, endIntervalTime=1701063280064000, executor=10.120.0.1, status=RUNNING, createTime=null, updateTime=null) 2023-11-27 13:36:40.071 INFO 10 --- [pool-scanner--1395-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:40.076 INFO 10 --- [pool-scanner--1393-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=24 2023-11-27 13:36:40.086 INFO 10 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2023-11-27T13:24:40.064, end=2023-11-27T13:34:40.064} 2023-11-27 13:36:40.087 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2023-11-27T13:05:40.052, end=2023-11-27T13:35:40.052} 2023-11-27 13:36:40.088 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000029, status:SUCCESS 2023-11-27 13:36:40.097 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000029, status:SUCCESS 2023-11-27 13:36:40.097 WARN 10 --- [pool-12-thread-1,,] o.p.s.d.i.OcpPerfSqlDiagnosisTaskService : OcpPerfSqlDiagnosisTaskEntity(taskName=SQL_PERF_DIAGNOSER, beginIntervalTime=1700793785688000, endIntervalTime=1700794086909000, executor=10.120.0.1, status=SUCCESS, createTime=2023-11-24T02:49:06.881Z, updateTime=2023-11-24T02:49:07.616Z) is too old, skip to diag OcpPerfSqlDiagnosisTaskEntity(taskName=SQL_PERF_DIAGNOSER, beginIntervalTime=1701061540097000, endIntervalTime=1701063340097000, executor=10.120.0.1, status=RUNNING, createTime=null, updateTime=null) 2023-11-27 13:36:40.105 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063400, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:40.105 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063400, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:40.106 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063400, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:40.106 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063400, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:40.108 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063400, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:40.114 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:40.116 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:40.118 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:40.118 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:40.119 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:40.125 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:40.126 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:40.128 INFO 10 --- [pool-scanner--1396-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:40.129 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-11-27 13:36:40.132 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2023-11-27T13:05:40.097, end=2023-11-27T13:35:40.097} 2023-11-27 13:36:40.133 INFO 10 --- [pool-scanner--1400-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:40.135 INFO 10 --- [pool-scanner--1399-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:40.142 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,fbb8a14d33be] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:40.147 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,fbb8a14d33be] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:40.148 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,fbb8a14d33be] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:40.148 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,fbb8a14d33be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:40.149 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,fbb8a14d33be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:40.160 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,fbb8a14d33be] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:40.160 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,fbb8a14d33be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:40.162 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,fbb8a14d33be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND timestamp > :startTime AND timestamp <= :endTime ORDER BY timestamp ASC LIMIT :limit OFFSET :offset, args: [2023-11-24 10:16:38.567361, 2023-11-27 13:36:32.779, 1000, 1000] 2023-11-27 13:36:40.171 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,fbb8a14d33be] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 2, startTime 2023-11-24 10:16:38.567361, endTime 2023-11-27 13:36:32.779 2023-11-27 13:36:40.182 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:40.188 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:40.192 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:40.192 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:40.193 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:40.199 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:40.200 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:40.202 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:36:40.202 INFO 10 --- [pool-subtask-executor21,554481c3fd1b426b,fbb8a14d33be] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000014, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:40.206 INFO 10 --- [pool-scanner--1407-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:40.207 INFO 10 --- [pool-scanner--1408-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:40.210 INFO 10 --- [pool-scanner--1398-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:40.217 INFO 10 --- [pool-scanner--1397-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:40.226 INFO 10 --- [pool-scanner--1409-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:40.234 INFO 10 --- [pool-scanner--1410-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:40.244 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:40.246 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:40.250 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:40.250 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:40.255 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:40.270 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:40.270 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:40.274 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:36:40.274 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-11-27 13:36:40.275 INFO 10 --- [pool-scanner--1406-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:40.288 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:40.288 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:40.291 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_1' and table_type='BASE TABLE' 2023-11-27 13:36:40.298 INFO 10 --- [pool-scanner--1405-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:40.299 INFO 10 --- [pool-scanner--1404-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:40.300 INFO 10 --- [pool-scanner--1414-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:40.322 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.120.0.1 2023-11-27 13:36:40.324 INFO 10 --- [pool-scanner--1403-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:40.335 INFO 10 --- [pool-scanner--1401-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:40.338 INFO 10 --- [pool-scanner--1415-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:40.343 INFO 10 --- [pool-scanner--1402-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:40.388 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:40.390 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:40.391 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:40.391 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:40.393 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:40.396 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:40.396 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:40.397 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-11-27 13:36:40.417 INFO 10 --- [pool-scanner--1413-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:40.427 INFO 10 --- [pool-scanner--1412-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:40.428 INFO 10 --- [pool-scanner--1419-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:40.448 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:40.458 INFO 10 --- [pool-scanner--1418-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:40.527 INFO 10 --- [pool-scanner--1416-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:40.542 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:40.542 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:40.542 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:40.543 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:40.546 INFO 10 --- [pool-scanner--1411-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:40.547 INFO 10 --- [pool-scanner--1420-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:40.550 INFO 10 --- [http-nio-0.0.0.0-8080-exec-4,2e82805f39a3466a,93837e8c192a] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.13, traceId=2e82805f39a3466a, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:36:40.553 INFO 10 --- [pool-scanner--1425-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:40.557 INFO 10 --- [pool-scanner--1423-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:40.557 INFO 10 --- [pool-scanner--1422-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:40.568 INFO 10 --- [pool-scanner--1427-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:40.572 INFO 10 --- [pool-scanner--1426-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:40.574 INFO 10 --- [pool-13-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:24:40.064, end=2023-11-27T13:34:40.064},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-11-27 13:36:40.578 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:40.578 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:40.579 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=5, name=daren, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1,zone3,zone4,zone5,zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.052, end=2023-11-27T13:35:40.052} 2023-11-27 13:36:40.582 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1] 2023-11-27 13:36:40.582 INFO 10 --- [pool-scanner--1430-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:40.590 INFO 10 --- [pool-scanner--1428-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:40.606 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_1 2023-11-27 13:36:40.606 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:36:40.614 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1;zone3;zone4;zone5;zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.097, end=2023-11-27T13:35:40.097} 2023-11-27 13:36:40.618 INFO 10 --- [pool-scanner--1429-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:40.620 INFO 10 --- [pool-scanner--1417-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:40.624 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_1, partition=p1701187200 2023-11-27 13:36:40.625 INFO 10 --- [pool-scanner--1421-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:40.628 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:40.628 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:40.632 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_1` add partition(partition p1701187200 values less than (1701187200)) 2023-11-27 13:36:40.635 INFO 10 --- [pool-13-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1, sqlAuditEnabled=null), right=null) 2023-11-27 13:36:40.643 INFO 10 --- [pool-scanner--1438-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:40.643 INFO 10 --- [pool-13-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1001, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:24:40.064, end=2023-11-27T13:34:40.064},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-11-27 13:36:40.643 INFO 10 --- [pool-13-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=meta_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1001, sqlAuditEnabled=null), right=null) 2023-11-27 13:36:40.646 INFO 10 --- [pool-scanner--1424-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:40.650 INFO 10 --- [pool-13-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1;zone3;zone4;zone5;zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:24:40.064, end=2023-11-27T13:34:40.064},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-11-27 13:36:40.650 INFO 10 --- [pool-13-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1;zone3;zone4;zone5;zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null) 2023-11-27 13:36:40.668 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=5, name=daren, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1,zone3,zone4,zone5,zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.052, end=2023-11-27T13:35:40.052} 2023-11-27 13:36:40.675 INFO 10 --- [pool-scanner--1437-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=24 2023-11-27 13:36:40.676 INFO 10 --- [pool-13-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 3 diagnose targets Pair(left=Tenant(id=5, name=daren, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1,zone3,zone4,zone5,zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:24:40.064, end=2023-11-27T13:34:40.064},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-11-27 13:36:40.686 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=5, name=daren, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1,zone3,zone4,zone5,zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.052, end=2023-11-27T13:35:40.052} 2023-11-27 13:36:40.720 INFO 10 --- [pool-scanner--1440-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:40.720 INFO 10 --- [pool-scanner--1434-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:40.723 INFO 10 --- [pool-scanner--1439-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:40.724 INFO 10 --- [pool-scanner--1436-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:40.727 INFO 10 --- [http-nio-0.0.0.0-8080-exec-4,2e82805f39a3466a,93837e8c192a] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.13, traceId=2e82805f39a3466a, duration=178 ms] 2023-11-27 13:36:40.753 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 5 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1;zone3;zone4;zone5;zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.097, end=2023-11-27T13:35:40.097} 2023-11-27 13:36:40.803 INFO 10 --- [pool-scanner--1441-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:40.804 INFO 10 --- [pool-scanner--1442-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:40.761 INFO 10 --- [pool-scanner--1433-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:40.810 INFO 10 --- [pool-scanner--1443-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:40.840 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-11-27 13:36:40.841 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:40.842 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:40.845 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1001] 2023-11-27 13:36:40.899 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:36:40.970 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-11-27 13:36:40.974 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:40.974 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:40.982 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1002] 2023-11-27 13:36:41.004 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:36:41.019 INFO 10 --- [pool-scanner--1446-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:40.767 INFO 10 --- [pool-scanner--1432-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:40.771 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=5, name=daren, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1,zone3,zone4,zone5,zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.052, end=2023-11-27T13:35:40.052} 2023-11-27 13:36:40.774 INFO 10 --- [pool-scanner--1431-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:41.053 INFO 10 --- [pool-scanner--1445-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:41.054 INFO 10 --- [pool-scanner--1448-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:41.075 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1;zone3;zone4;zone5;zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.052, end=2023-11-27T13:35:40.052} 2023-11-27 13:36:41.092 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-11-27 13:36:41.092 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-11-27 13:36:41.092 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=obcluster, obClusterId=1 2023-11-27 13:36:41.109 INFO 10 --- [pool-scanner--1449-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:41.109 INFO 10 --- [pool-scanner--1451-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:41.124 INFO 10 --- [pool-13-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=daren, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1,zone3,zone4,zone5,zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null) 2023-11-27 13:36:41.159 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1;zone3;zone4;zone5;zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.052, end=2023-11-27T13:35:40.052} 2023-11-27 13:36:41.161 INFO 10 --- [pool-13-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:24:40.064, end=2023-11-27T13:34:40.064},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-11-27 13:36:41.161 INFO 10 --- [pool-13-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null) 2023-11-27 13:36:41.161 INFO 10 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2023-11-27 13:36:41.172 INFO 10 --- [pool-scanner--1454-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:41.176 INFO 10 --- [pool-scanner--1447-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:41.183 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1;zone3;zone4;zone5;zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.052, end=2023-11-27T13:35:40.052} 2023-11-27 13:36:41.186 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,7022e8225190] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000034, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:41.190 INFO 10 --- [pool-scanner--1452-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:41.202 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1;zone3;zone4;zone5;zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.052, end=2023-11-27T13:35:40.052} 2023-11-27 13:36:41.212 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.052, end=2023-11-27T13:35:40.052} 2023-11-27 13:36:40.788 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:40.789 INFO 10 --- [pool-scanner--1435-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=24 2023-11-27 13:36:40.791 INFO 10 --- [pool-scanner--1444-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:41.217 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:41.218 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:41.218 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:41.222 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:41.225 INFO 10 --- [pool-scanner--1450-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:41.228 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:41.228 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:41.230 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:36:41.231 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.052, end=2023-11-27T13:35:40.052} 2023-11-27 13:36:41.234 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-11-27 13:36:41.234 INFO 10 --- [pool-scanner--1455-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:41.244 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.052, end=2023-11-27T13:35:40.052} 2023-11-27 13:36:41.285 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.052, end=2023-11-27T13:35:40.052} 2023-11-27 13:36:41.292 INFO 10 --- [pool-scanner--1453-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:41.293 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1001, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.052, end=2023-11-27T13:35:40.052} 2023-11-27 13:36:41.321 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.120.0.1 2023-11-27 13:36:41.322 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-11-27 13:36:41.329 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1001, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.052, end=2023-11-27T13:35:40.052} 2023-11-27 13:36:41.333 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1001, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.052, end=2023-11-27T13:35:40.052} 2023-11-27 13:36:41.346 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1001, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.052, end=2023-11-27T13:35:40.052} 2023-11-27 13:36:41.352 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.052, end=2023-11-27T13:35:40.052} 2023-11-27 13:36:41.356 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-11-27 13:36:41.357 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.052, end=2023-11-27T13:35:40.052} 2023-11-27 13:36:41.363 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.052, end=2023-11-27T13:35:40.052} 2023-11-27 13:36:41.381 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.052, end=2023-11-27T13:35:40.052} 2023-11-27 13:36:41.381 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2023-11-27 13:36:41.415 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5] 2023-11-27 13:36:41.420 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5] 2023-11-27 13:36:41.466 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1;zone3;zone4;zone5;zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.097, end=2023-11-27T13:35:40.097} 2023-11-27 13:36:41.564 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 24 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1;zone3;zone4;zone5;zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.097, end=2023-11-27T13:35:40.097} 2023-11-27 13:36:41.596 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:41.596 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:41.608 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_1' and table_type='BASE TABLE' 2023-11-27 13:36:41.662 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.10:62888/metrics/stat, ret=1 2023-11-27 13:36:41.711 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.10:62889/metrics/node/host, ret=1 2023-11-27 13:36:41.728 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000009, status:SUCCESS 2023-11-27 13:36:41.747 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000009, status:SUCCESS 2023-11-27 13:36:41.755 INFO 10 --- [pool-scanner--1474-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:41.755 INFO 10 --- [pool-scanner--1475-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:41.755 INFO 10 --- [pool-scanner--1476-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:41.756 INFO 10 --- [pool-scanner--1463-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:41.758 INFO 10 --- [pool-scanner--1462-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:41.782 INFO 10 --- [pool-scanner--1461-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:41.785 INFO 10 --- [pool-scanner--1460-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:41.796 INFO 10 --- [pool-overview-container-6,,e0803845ee1f] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-11-27 13:36:41.797 INFO 10 --- [pool-overview-container-6,,e0803845ee1f] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:36:41.808 INFO 10 --- [pool-scanner--1478-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:41.808 INFO 10 --- [pool-scanner--1459-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:41.808 INFO 10 --- [pool-scanner--1479-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:41.829 INFO 10 --- [pool-scanner--1458-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:41.867 INFO 10 --- [pool-scanner--1491-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:41.875 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-11-27 13:36:41.880 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_1 2023-11-27 13:36:41.887 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_1, partition=p1701273600 2023-11-27 13:36:41.888 INFO 10 --- [pool-scanner--1457-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:41.888 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.097, end=2023-11-27T13:35:40.097} 2023-11-27 13:36:41.901 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:41.901 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:41.908 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.097, end=2023-11-27T13:35:40.097} 2023-11-27 13:36:41.911 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_1` add partition(partition p1701273600 values less than (1701273600)) 2023-11-27 13:36:41.922 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.10:62889/metrics/stat, ret=1 2023-11-27 13:36:41.937 INFO 10 --- [pool-scanner--1492-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:41.940 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.097, end=2023-11-27T13:35:40.097} 2023-11-27 13:36:41.966 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.097, end=2023-11-27T13:35:40.097} 2023-11-27 13:36:41.982 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.10:62889/metrics/node/obproxy, ret=1 2023-11-27 13:36:42.264 INFO 10 --- [pool-scanner--1493-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:42.270 INFO 10 --- [pool-scanner--1470-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:42.271 INFO 10 --- [pool-scanner--1499-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:42.272 INFO 10 --- [pool-scanner--1498-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:42.276 INFO 10 --- [pool-scanner--1472-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:42.276 INFO 10 --- [pool-scanner--1500-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:42.276 INFO 10 --- [pool-scanner--1469-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:42.277 INFO 10 --- [pool-scanner--1473-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:42.277 INFO 10 --- [pool-scanner--1468-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:42.277 INFO 10 --- [pool-scanner--1465-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:42.277 INFO 10 --- [pool-scanner--1467-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:42.277 INFO 10 --- [pool-scanner--1466-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:42.277 INFO 10 --- [pool-scanner--1464-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:42.281 INFO 10 --- [pool-scanner--1496-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:42.282 INFO 10 --- [pool-scanner--1471-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:42.283 INFO 10 --- [pool-scanner--1495-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:42.284 INFO 10 --- [pool-scanner--1483-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=24 2023-11-27 13:36:42.287 INFO 10 --- [pool-scanner--1477-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:42.288 INFO 10 --- [pool-scanner--1484-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:42.288 INFO 10 --- [pool-scanner--1485-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:42.291 INFO 10 --- [pool-scanner--1486-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:42.291 INFO 10 --- [pool-scanner--1487-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:42.291 INFO 10 --- [pool-scanner--1497-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:42.291 INFO 10 --- [http-nio-0.0.0.0-8080-exec-8,7ed5131701c3498c,91d502c53239] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.12, traceId=7ed5131701c3498c, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:36:42.291 INFO 10 --- [pool-scanner--1482-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=24 2023-11-27 13:36:42.293 INFO 10 --- [pool-scanner--1494-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:42.293 INFO 10 --- [pool-scanner--1490-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:42.295 INFO 10 --- [pool-scanner--1488-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:42.295 INFO 10 --- [pool-scanner--1489-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:42.295 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-11-27 13:36:42.296 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:42.302 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:42.302 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:42.302 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:42.303 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_1' and table_type='BASE TABLE' 2023-11-27 13:36:42.305 INFO 10 --- [pool-scanner--1503-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:42.308 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.097, end=2023-11-27T13:35:40.097} 2023-11-27 13:36:42.311 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:42.311 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:42.312 INFO 10 --- [pool-scanner--1501-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:42.312 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:42.316 INFO 10 --- [pool-scanner--1456-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:42.316 INFO 10 --- [pool-scanner--1504-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:42.321 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.097, end=2023-11-27T13:35:40.097} 2023-11-27 13:36:42.330 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:36:42.332 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.097, end=2023-11-27T13:35:40.097} 2023-11-27 13:36:42.334 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:36:42.335 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:42.335 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:42.336 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-11-27 13:36:42.347 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.097, end=2023-11-27T13:35:40.097} 2023-11-27 13:36:42.348 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-11-27 13:36:42.358 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:36:42.366 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=5, name=daren, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1,zone3,zone4,zone5,zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.097, end=2023-11-27T13:35:40.097} 2023-11-27 13:36:42.383 INFO 10 --- [pool-scanner--1502-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:42.383 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 7 diagnose targets Pair(left=Tenant(id=5, name=daren, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1,zone3,zone4,zone5,zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.097, end=2023-11-27T13:35:40.097} 2023-11-27 13:36:42.387 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-11-27 13:36:42.288 INFO 10 --- [pool-scanner--1480-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:42.288 INFO 10 --- [pool-scanner--1481-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:42.394 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:42.394 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:42.396 INFO 10 --- [http-nio-0.0.0.0-8080-exec-8,7ed5131701c3498c,91d502c53239] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.12, traceId=7ed5131701c3498c, duration=106 ms] 2023-11-27 13:36:42.399 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-11-27 13:36:42.400 INFO 10 --- [pool-scanner--1513-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:42.414 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.10:62889/metrics/obproxy, ret=1 2023-11-27 13:36:42.433 INFO 10 --- [pool-scanner--1515-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:42.440 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:36:42.443 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.11:62889/metrics/node/host, ret=1 2023-11-27 13:36:42.454 INFO 10 --- [pool-scanner--1510-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:42.489 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-11-27 13:36:42.489 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-11-27 13:36:42.489 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=darenOB, obClusterId=10 2023-11-27 13:36:42.490 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_1 2023-11-27 13:36:42.502 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f0d6bb37dec5] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000035, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:42.505 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.11:62888/metrics/stat, ret=1 2023-11-27 13:36:42.509 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ocp_metric_data_1, partition=p1700150400 2023-11-27 13:36:42.510 INFO 10 --- [pool-scanner--1511-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:42.511 INFO 10 --- [pool-scanner--1517-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:42.511 INFO 10 --- [pool-scanner--1512-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:42.511 INFO 10 --- [pool-scanner--1509-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:42.512 INFO 10 --- [pool-scanner--1516-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:42.513 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:42.513 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:42.522 INFO 10 --- [pool-scanner--1508-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:42.524 INFO 10 --- [pool-scanner--1507-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=24 2023-11-27 13:36:42.553 INFO 10 --- [pool-scanner--1505-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:42.554 INFO 10 --- [pool-scanner--1518-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=132 2023-11-27 13:36:42.561 INFO 10 --- [pool-scanner--1514-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:42.561 INFO 10 --- [pool-scanner--1519-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:42.565 INFO 10 --- [pool-scanner--1520-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:42.579 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=5, name=daren, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1,zone3,zone4,zone5,zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.097, end=2023-11-27T13:35:40.097} 2023-11-27 13:36:42.594 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 11 diagnose targets Pair(left=Tenant(id=5, name=daren, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1,zone3,zone4,zone5,zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.097, end=2023-11-27T13:35:40.097} 2023-11-27 13:36:42.600 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.a.i.operator.SysStatOperator : ob_hist_sysstat snapshot generate data success, clusterName = obcluster, snapshotId = 8140, endIntervalTime = 1701063366973000 2023-11-27 13:36:42.604 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_1` drop partition (p1700150400) 2023-11-27 13:36:42.606 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.11:62889/metrics/stat, ret=1 2023-11-27 13:36:42.607 INFO 10 --- [pool-scanner--1506-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=24 2023-11-27 13:36:42.618 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.a.i.operator.SysStatOperator : ob_hist_sysstat snapshot generate data success, clusterName = obcluster, snapshotId = 8140, endIntervalTime = 1701063366973000 2023-11-27 13:36:42.618 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for SysStatSnapshotOperator, error count is 1 2023-11-27 13:36:42.618 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for ServerInfoSnapshotOperator 2023-11-27 13:36:42.618 WARN 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid time to generate snapshot data, snapshot end interval time is 1701063366973000, now is 1701063402618000 2023-11-27 13:36:42.637 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.11:62889/metrics/node/obproxy, ret=1 2023-11-27 13:36:42.651 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.a.i.operator.SysStatOperator : ob_hist_sysstat snapshot generate data success, clusterName = darenOB, snapshotId = 8139, endIntervalTime = 1701063366973000 2023-11-27 13:36:42.651 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.a.i.operator.SysStatOperator : ob_hist_sysstat snapshot generate data success, clusterName = darenOB, snapshotId = 8139, endIntervalTime = 1701063366973000 2023-11-27 13:36:42.651 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for SysStatSnapshotOperator, error count is 0 2023-11-27 13:36:42.651 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for ServerInfoSnapshotOperator 2023-11-27 13:36:42.651 WARN 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid time to generate snapshot data, snapshot end interval time is 1701063366973000, now is 1701063402651000 2023-11-27 13:36:42.654 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.11:62889/metrics/obproxy, ret=1 2023-11-27 13:36:42.679 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.12:62889/metrics/node/host, ret=1 2023-11-27 13:36:42.702 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.12:62888/metrics/stat, ret=1 2023-11-27 13:36:42.767 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:42.769 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:42.773 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.12:62889/metrics/stat, ret=1 2023-11-27 13:36:42.775 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:42.775 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:42.779 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:42.779 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:42.779 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:42.779 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:42.780 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:42.789 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:42.794 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.12:62889/metrics/obproxy, ret=1 2023-11-27 13:36:42.797 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:42.798 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:42.799 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ read_consistency(weak) */ svr_ip, svr_port, zone, leader_count, unit_num, cpu_capacity, cpu_total AS cpu_capacity_max, cpu_assigned, cpu_max_assigned AS cpu_assigned_max, mem_capacity, mem_total AS mem_capacity_max, mem_assigned, mem_max_assigned AS mem_assigned_max FROM oceanbase.__all_virtual_server_stat 2023-11-27 13:36:42.804 INFO 10 --- [http-nio-0.0.0.0-8080-exec-6,894dd8c68d2b462a,4c0b5f6ce73b] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.12, traceId=894dd8c68d2b462a, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:36:42.811 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:42.812 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:42.814 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, DATA_DISK_CAPACITY AS DISK_CAPACITY, DATA_DISK_IN_USE AS DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2023-11-27 13:36:42.814 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.12:62889/metrics/node/obproxy, ret=1 2023-11-27 13:36:42.870 INFO 10 --- [pool-sql_perf_diagnoser7,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:42.871 INFO 10 --- [pool-sql_perf_diagnoser7,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:42.871 INFO 10 --- [pool-sql_perf_diagnoser7,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:42.872 INFO 10 --- [pool-sql_perf_diagnoser7,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:42.884 INFO 10 --- [pool-sql_perf_diagnoser7,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:42.884 INFO 10 --- [pool-sql_perf_diagnoser7,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:42.886 INFO 10 --- [pool-sql_perf_diagnoser7,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT/*+ QUERY_TIMEOUT(30000000) */ con_id as tenant_id, object_id as database_id, object_name as database_name FROM oceanbase.cdb_objects where con_id = ? and OBJECT_TYPE = 'DATABASE' , args: [1002] 2023-11-27 13:36:42.906 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.13:62889/metrics/node/host, ret=1 2023-11-27 13:36:42.980 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:42.981 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:42.981 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ read_consistency(weak) */ svr_ip, svr_port, total_size AS disk_capacity, total_size - free_size AS disk_in_use FROM oceanbase.__all_virtual_disk_stat 2023-11-27 13:36:43.006 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.13:62888/metrics/stat, ret=1 2023-11-27 13:36:43.114 INFO 10 --- [http-nio-0.0.0.0-8080-exec-6,894dd8c68d2b462a,4c0b5f6ce73b] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.12, traceId=894dd8c68d2b462a, duration=311 ms] 2023-11-27 13:36:43.128 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.13:62889/metrics/stat, ret=1 2023-11-27 13:36:43.131 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:36:43.129618+08:00, lastProcessTime=2023-11-27T13:36:34.549027+08:00 2023-11-27 13:36:43.142 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.13:62889/metrics/node/obproxy, ret=1 2023-11-27 13:36:43.162 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.13:62889/metrics/obproxy, ret=1 2023-11-27 13:36:43.179 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:43.179 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:43.180 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_1' and table_type='BASE TABLE' 2023-11-27 13:36:43.191 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ocp_metric_data_1, partition=p1700236800 2023-11-27 13:36:43.196 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:43.197 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:43.198 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_1` drop partition (p1700236800) 2023-11-27 13:36:43.332 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:43.332 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:43.335 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_1' and table_type='BASE TABLE' 2023-11-27 13:36:43.347 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ocp_metric_data_1, partition=p1700323200 2023-11-27 13:36:43.351 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:43.352 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:43.354 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_1` drop partition (p1700323200) 2023-11-27 13:36:43.357 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.a.i.operator.ServerInfoOperator : ob_hist_server snapshot generate data success, clusterName = obcluster, snapshotId = 8140, endIntervalTime = 1701063366973000 2023-11-27 13:36:43.357 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for ServerInfoSnapshotOperator, error count is 1 2023-11-27 13:36:43.357 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for PartitionHitOperator 2023-11-27 13:36:43.357 WARN 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid time to generate snapshot data, snapshot end interval time is 1701063366973000, now is 1701063403357000 2023-11-27 13:36:43.443 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.a.i.operator.ServerInfoOperator : ob_hist_server snapshot generate data success, clusterName = darenOB, snapshotId = 8139, endIntervalTime = 1701063366973000 2023-11-27 13:36:43.443 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for ServerInfoSnapshotOperator, error count is 0 2023-11-27 13:36:43.443 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for PartitionHitOperator 2023-11-27 13:36:43.491 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-11-27 13:36:43.443 WARN 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid time to generate snapshot data, snapshot end interval time is 1701063366973000, now is 1701063403443000 2023-11-27 13:36:43.504 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-11-27 13:36:43.515 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:43.515 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:43.516 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_1' and table_type='BASE TABLE' 2023-11-27 13:36:43.533 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:43.534 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:43.538 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:43.541 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:43.541 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:43.542 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2023-11-27 13:36:43.609 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5] 2023-11-27 13:36:43.623 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1701100800 2023-11-27 13:36:43.626 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5] 2023-11-27 13:36:43.627 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:43.627 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:43.629 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1701100800 values less than (1701100800)) 2023-11-27 13:36:43.632 INFO 10 --- [pool-ocp-schedules-19,,db7f8cf11ad1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:36:43.640 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : groupAggregate, now=2023-11-27T13:36:43.129618+08:00, groupByKey=0:no_enough_exporter:1, alarmsToSentCount=4, alarmIds=[2000004, 2000002, 2000003, 2000001] 2023-11-27 13:36:43.640 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : createNotification start, groupByKey=0:no_enough_exporter:1 2023-11-27 13:36:43.641 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : createNotification done, groupByKey=0:no_enough_exporter:1, alarmIds=[2000004, 2000002, 2000003, 2000001] 2023-11-27 13:36:43.641 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.ChannelAggregator : aggregate done, channelId=100000, now=2023-11-27T13:36:43.129618+08:00, resultsCount=1 2023-11-27 13:36:43.641 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : aggregate done, channelId=100000, alarmsCount=0, resultsCount=1 2023-11-27 13:36:43.641 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.ChannelAggregator : groupAggregate markSendSuccess, groupByKey=0:no_enough_exporter:1, now=2023-11-27T13:36:43.129618+08:00, alarmIds=[2000004, 2000002, 2000003, 2000001] 2023-11-27 13:36:43.641 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:36:43.129618+08:00 2023-11-27 13:36:43.682 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000028, status:SUCCESS 2023-11-27 13:36:43.692 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000028, status:SUCCESS 2023-11-27 13:36:43.724 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.10:62889/metrics/ob/extra, ret=1 2023-11-27 13:36:43.747 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.10:62889/metrics/node/ob, ret=1 2023-11-27 13:36:43.769 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.10:62889/metrics/ob/basic, ret=1 2023-11-27 13:36:43.926 INFO 10 --- [pool-scanner--1521-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:43.936 INFO 10 --- [pool-scanner--1524-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:43.937 INFO 10 --- [pool-scanner--1523-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:43.938 INFO 10 --- [pool-scanner--1522-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:43.940 INFO 10 --- [pool-scanner--1526-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:43.942 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for PartitionHitOperator, error count is 1 2023-11-27 13:36:43.942 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for RpcSnapshotOperator 2023-11-27 13:36:43.942 WARN 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid time to generate snapshot data, snapshot end interval time is 1701063366973000, now is 1701063403942000 2023-11-27 13:36:43.959 INFO 10 --- [pool-scanner--1525-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:43.962 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for PartitionHitOperator, error count is 0 2023-11-27 13:36:43.962 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for RpcSnapshotOperator 2023-11-27 13:36:43.962 WARN 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid time to generate snapshot data, snapshot end interval time is 1701063366973000, now is 1701063403962000 2023-11-27 13:36:43.998 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:43.999 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:43.999 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:43.999 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:44.000 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:44.009 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:44.009 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:44.010 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select a.tenant_id as ob_tenant_id, a.svr_ip as server_ip, a.pcode as packet_code, a.pcode_name as packet_name, a.count as outgoing_count, a.total_size as outgoing_size_byte, b.count as incoming_count, b.total_size as incoming_size_byte, a.failure as outgoing_fail_count, a.timeout as outgoing_timeout_count from gv$ob_rpc_outgoing a, gv$ob_rpc_incoming b where a.svr_ip=b.svr_ip and a.pcode=b.pcode and a.tenant_id=b.tenant_id and a.count > 0 and b.count>0 2023-11-27 13:36:44.012 INFO 10 --- [pool-sql_perf_diagnoser7,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:44.014 INFO 10 --- [pool-sql_perf_diagnoser7,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:44.021 INFO 10 --- [pool-sql_perf_diagnoser7,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:44.021 INFO 10 --- [pool-sql_perf_diagnoser7,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:44.022 INFO 10 --- [pool-sql_perf_diagnoser7,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:44.026 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:44.028 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:44.029 INFO 10 --- [pool-sql_perf_diagnoser7,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:44.029 INFO 10 --- [pool-sql_perf_diagnoser7,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:44.030 INFO 10 --- [pool-sql_perf_diagnoser7,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-11-27 13:36:44.035 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:44.035 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:44.037 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:44.057 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:44.057 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:44.057 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:44.057 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:44.067 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2023-11-27 13:36:44.067 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select a.tenant_id as ob_tenant_id, a.ip as server_ip, a.pcode as packet_code, a.pcode_name as packet_name, a.count as outgoing_count, a.total_size as outgoing_size_byte, b.count as incoming_count, b.total_size as incoming_size_byte, a.failure as outgoing_fail_count, a.timeout as outgoing_timeout_count from gv$obrpc_outgoing a, gv$obrpc_incoming b where a.ip=b.ip and a.pcode=b.pcode and a.tenant_id=b.tenant_id and a.count > 0 and b.count>0 2023-11-27 13:36:44.156 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-11-27 13:36:44.196 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.11:62889/metrics/ob/extra, ret=1 2023-11-27 13:36:44.197 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1001, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.097, end=2023-11-27T13:35:40.097} 2023-11-27 13:36:44.209 WARN 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.a.internal.operator.RpcOperator : Invalid timestamp for snapshot. last snapshot is ObHistSnapshotEntity(clusterId=2, obClusterId=10, clusterName=darenOB, snapshotId=8139, version=0, beginIntervalTime=1701059766973000, endIntervalTime=1701063366973000, taskInstanceId=8000005, flushElapsed=0, errorCount=0, errorInfo=TIME_ERROR, createTime=null, updateTime=null), current snapshot context is GenerateSnapshotContext{clusterEntity=ObClusterEntity{id=2, name=darenOB, obClusterId=10, status=RUNNING}, snapshotId=8139, beginIntervalTime=1701059766973000, beginIntervalTimeString=2023-11-27T12:36:06.973, endIntervalTime=1701063366973000, endIntervalTimeString=2023-11-27T13:36:06.973} 2023-11-27 13:36:44.235 WARN 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.a.internal.operator.RpcOperator : Invalid timestamp for snapshot. last snapshot is ObHistSnapshotEntity(clusterId=1, obClusterId=1, clusterName=obcluster, snapshotId=8140, version=0, beginIntervalTime=1701059766973000, endIntervalTime=1701063366973000, taskInstanceId=8000005, flushElapsed=0, errorCount=1, errorInfo=TIME_ERROR SERVER_RESTART, createTime=null, updateTime=null), current snapshot context is GenerateSnapshotContext{clusterEntity=ObClusterEntity{id=1, name=obcluster, obClusterId=1, status=RUNNING}, snapshotId=8140, beginIntervalTime=1701059766973000, beginIntervalTimeString=2023-11-27T12:36:06.973, endIntervalTime=1701063366973000, endIntervalTimeString=2023-11-27T13:36:06.973} 2023-11-27 13:36:44.239 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:36:44.270 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:36:44.274 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:36:44.290 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.11:62889/metrics/ob/basic, ret=1 2023-11-27 13:36:44.335 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1001, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.097, end=2023-11-27T13:35:40.097} 2023-11-27 13:36:44.370 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1001, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.097, end=2023-11-27T13:35:40.097} 2023-11-27 13:36:44.387 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1001, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:05:40.097, end=2023-11-27T13:35:40.097} 2023-11-27 13:36:44.387 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-11-27 13:36:44.387 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2023-11-27 13:36:44.421 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2023-11-27 13:36:44.438 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1701187200 2023-11-27 13:36:44.441 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:44.441 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:44.446 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1701187200 values less than (1701187200)) 2023-11-27 13:36:44.676 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.11:62889/metrics/node/ob, ret=1 2023-11-27 13:36:44.694 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.12:62889/metrics/ob/basic, ret=1 2023-11-27 13:36:44.743 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:36:44.743 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:36:44.903 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:44.903 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:44.906 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2023-11-27 13:36:44.993 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2023-11-27 13:36:45.000 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1701273600 2023-11-27 13:36:45.011 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:45.011 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:45.016 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1701273600 values less than (1701273600)) 2023-11-27 13:36:45.048 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-11-27 13:36:45.062 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.12:62889/metrics/ob/extra, ret=1 2023-11-27 13:36:45.104 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063405, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:45.108 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063405, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:45.117 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063405, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:45.144 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063405, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:45.150 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063405, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:45.177 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5] 2023-11-27 13:36:45.177 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.12:62889/metrics/node/ob, ret=1 2023-11-27 13:36:45.228 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.13:62889/metrics/ob/basic, ret=1 2023-11-27 13:36:45.238 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.13:62889/metrics/node/ob, ret=1 2023-11-27 13:36:45.264 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,07ed6be6b585] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000028, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:45.334 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:36:45.338 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:36:45.364 INFO 10 --- [pool-scanner--1528-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:45.368 INFO 10 --- [pool-scanner--1527-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:45.371 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:45.371 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:45.372 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2023-11-27 13:36:45.377 INFO 10 --- [pool-scanner--1529-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:45.390 INFO 10 --- [pool-scanner--1530-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:45.390 INFO 10 --- [pool-scanner--1531-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:45.394 INFO 10 --- [pool-scanner--1532-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:45.396 INFO 10 --- [pool-scanner--1533-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:45.435 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-11-27 13:36:45.443 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5] 2023-11-27 13:36:45.458 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2023-11-27 13:36:45.460 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1698163200 2023-11-27 13:36:45.461 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:45.461 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:45.462 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` drop partition (p1698163200) 2023-11-27 13:36:45.490 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:36:45.500 INFO 10 --- [pool-scanner--1534-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:45.530 INFO 10 --- [pool-scanner--1536-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:45.530 INFO 10 --- [pool-scanner--1535-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:45.549 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:45.549 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:45.551 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2023-11-27 13:36:45.572 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1698249600 2023-11-27 13:36:45.574 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:45.574 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:45.577 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` drop partition (p1698249600) 2023-11-27 13:36:45.596 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : active exporter, instance=http://133.197.204.13:62889/metrics/ob/extra, ret=1 2023-11-27 13:36:45.600 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-11-27 13:36:45.603 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/ob/extra, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:36:45.603 WARN 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/ob/extra is inactive 2023-11-27 13:36:45.607 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/node/ob, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:36:45.607 WARN 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/node/ob is inactive 2023-11-27 13:36:45.613 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/ob/basic, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:36:45.613 WARN 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/ob/basic is inactive 2023-11-27 13:36:45.654 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:45.654 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:45.655 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2023-11-27 13:36:45.674 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1698336000 2023-11-27 13:36:45.675 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:45.675 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:45.675 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` drop partition (p1698336000) 2023-11-27 13:36:45.684 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5] 2023-11-27 13:36:45.695 INFO 10 --- [pool-prometheus-exporter-check2,,07dabffcceea] .o.m.s.OcpExporterStatusCheckServiceImpl : checkInactive found active exporter(s), count=32 2023-11-27 13:36:45.831 INFO 10 --- [pool-scanner--1538-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:45.831 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:45.832 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:45.836 INFO 10 --- [pool-scanner--1537-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:45.847 INFO 10 --- [pool-scanner--1539-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:45.856 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2023-11-27 13:36:45.892 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:45.892 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:45.896 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:45.900 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:45.900 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:45.905 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2023-11-27 13:36:45.933 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-11-27 13:36:45.976 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:45.976 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:45.977 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:45.978 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:45.978 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:45.985 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2023-11-27 13:36:46.012 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5] 2023-11-27 13:36:46.026 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:46.026 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:46.027 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:46.030 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:46.030 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:46.037 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2023-11-27 13:36:46.065 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000022, status:SUCCESS 2023-11-27 13:36:46.067 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000022, status:SUCCESS 2023-11-27 13:36:46.071 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:36:46.131 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_second, partition=p20231128 2023-11-27 13:36:46.146 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:46.147 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:46.148 INFO 10 --- [pool-scanner--1540-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=6 2023-11-27 13:36:46.151 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_second` add partition(partition p20231128 values less than (1701100800)) 2023-11-27 13:36:46.181 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for RpcSnapshotOperator, error count is 1 2023-11-27 13:36:46.181 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for MemoryOperator 2023-11-27 13:36:46.181 WARN 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid time to generate snapshot data, snapshot end interval time is 1701063366973000, now is 1701063406181000 2023-11-27 13:36:46.187 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-11-27 13:36:46.223 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5] 2023-11-27 13:36:46.399 INFO 10 --- [pool-scanner--1541-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=246 2023-11-27 13:36:46.423 INFO 10 --- [pool-scanner--1542-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=246 2023-11-27 13:36:46.437 INFO 10 --- [pool-scanner--1543-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=246 2023-11-27 13:36:46.481 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for MemoryOperator, error count is 1 2023-11-27 13:36:46.481 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for SystemEventSnapshotOperator 2023-11-27 13:36:46.481 WARN 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid time to generate snapshot data, snapshot end interval time is 1701063366973000, now is 1701063406481000 2023-11-27 13:36:46.528 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-11-27 13:36:46.649 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5] 2023-11-27 13:36:46.815 INFO 10 --- [pool-scanner--1544-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=1050 2023-11-27 13:36:46.819 INFO 10 --- [pool-scanner--1546-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=1050 2023-11-27 13:36:46.898 INFO 10 --- [pool-scanner--1545-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=1050 2023-11-27 13:36:46.900 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.a.i.o.SystemEventOperator : [System event] store 0 targets 2023-11-27 13:36:46.900 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for SystemEventSnapshotOperator, error count is 1 2023-11-27 13:36:46.901 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate all snapshot, failCount = 1, beans = [sqlAuditSnapshotOperator, CacheSnapshotOperator, LatchSnapshotOperator, ParameterSnapshotOperator, RowLockSnapshotOperator, ClusterInfoSnapshotOperator, TenantInfoSnapshotOperator, ReplicaInfoOperator, SysStatSnapshotOperator, ServerInfoSnapshotOperator, PartitionHitOperator, RpcSnapshotOperator, MemoryOperator, SystemEventSnapshotOperator], fail beans is , fail info is 2023-11-27 13:36:47.066 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,39b3baa46685] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000006, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:36:47.074 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-11-27 13:36:47.084 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5] 2023-11-27 13:36:47.123 INFO 10 --- [pool-scanner--1549-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:47.123 INFO 10 --- [pool-scanner--1547-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:47.125 INFO 10 --- [pool-scanner--1548-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:47.129 INFO 10 --- [pool-scanner--1551-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:47.129 INFO 10 --- [pool-scanner--1550-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:47.130 INFO 10 --- [pool-scanner--1552-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:47.132 INFO 10 --- [pool-scanner--1553-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=46 2023-11-27 13:36:47.143 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-11-27 13:36:47.158 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5] 2023-11-27 13:36:47.189 INFO 10 --- [pool-scanner--1556-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=24 2023-11-27 13:36:47.189 INFO 10 --- [pool-scanner--1555-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=24 2023-11-27 13:36:47.190 INFO 10 --- [pool-scanner--1554-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=24 2023-11-27 13:36:47.197 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-11-27 13:36:47.203 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5] 2023-11-27 13:36:47.225 INFO 10 --- [pool-scanner--1558-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=24 2023-11-27 13:36:47.225 INFO 10 --- [pool-scanner--1559-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=24 2023-11-27 13:36:47.225 INFO 10 --- [pool-scanner--1557-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=24 2023-11-27 13:36:47.238 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-11-27 13:36:47.241 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5] 2023-11-27 13:36:47.320 INFO 10 --- [pool-scanner--1560-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=24 2023-11-27 13:36:47.324 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for RpcSnapshotOperator, error count is 0 2023-11-27 13:36:47.324 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for MemoryOperator 2023-11-27 13:36:47.324 WARN 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid time to generate snapshot data, snapshot end interval time is 1701063366973000, now is 1701063407324000 2023-11-27 13:36:47.328 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] slow query, durationMillis=1177, sql=alter table `metric_data_second` add partition(partition p20231128 values less than (1701100800)) 2023-11-27 13:36:47.331 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:47.331 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:47.332 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_second' and table_type='BASE TABLE' 2023-11-27 13:36:47.333 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-11-27 13:36:47.337 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5] 2023-11-27 13:36:47.469 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_second 2023-11-27 13:36:47.479 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_second, partition=p20231129 2023-11-27 13:36:47.480 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:47.481 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:47.482 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_second` add partition(partition p20231129 values less than (1701187200)) 2023-11-27 13:36:47.704 INFO 10 --- [pool-scanner--1563-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=2982 2023-11-27 13:36:47.759 INFO 10 --- [pool-scanner--1562-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=2982 2023-11-27 13:36:47.872 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:36:47.928 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:47.929 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:47.938 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_second' and table_type='BASE TABLE' 2023-11-27 13:36:47.953 INFO 10 --- [pool-scanner--1561-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=2982 2023-11-27 13:36:47.956 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for MemoryOperator, error count is 0 2023-11-27 13:36:47.956 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for SystemEventSnapshotOperator 2023-11-27 13:36:47.956 WARN 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Invalid time to generate snapshot data, snapshot end interval time is 1701063366973000, now is 1701063407956000 2023-11-27 13:36:47.962 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-11-27 13:36:47.966 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5] 2023-11-27 13:36:48.038 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_second 2023-11-27 13:36:48.051 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_second, partition=p20231130 2023-11-27 13:36:48.066 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:48.067 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:48.069 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_second` add partition(partition p20231130 values less than (1701273600)) 2023-11-27 13:36:48.339 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:36:48.343 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:36:49.062 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:49.063 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:49.064 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_second' and table_type='BASE TABLE' 2023-11-27 13:36:49.293 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_second 2023-11-27 13:36:49.297 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=metric_data_second, partition=p20231117 2023-11-27 13:36:49.299 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:49.299 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:49.303 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_second` drop partition (p20231117) 2023-11-27 13:36:49.541 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:49.542 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:49.543 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_second' and table_type='BASE TABLE' 2023-11-27 13:36:49.609 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=metric_data_second, partition=p20231118 2023-11-27 13:36:49.611 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:49.612 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:49.621 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_second` drop partition (p20231118) 2023-11-27 13:36:49.733 INFO 10 --- [http-nio-0.0.0.0-8080-exec-9,f05a7773875749a8,af4bdf573d13] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=133.197.206.97, traceId=f05a7773875749a8, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-11-27 13:36:49.747 INFO 10 --- [http-nio-0.0.0.0-8080-exec-9,f05a7773875749a8,af4bdf573d13] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.206.97, traceId=f05a7773875749a8, duration=15 ms] 2023-11-27 13:36:49.797 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:49.798 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:49.800 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_second' and table_type='BASE TABLE' 2023-11-27 13:36:49.819 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=metric_data_second, partition=p20231119 2023-11-27 13:36:49.821 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:49.822 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:49.828 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_second` drop partition (p20231119) 2023-11-27 13:36:49.955 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:49.956 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:49.957 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_second' and table_type='BASE TABLE' 2023-11-27 13:36:49.981 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:49.982 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:49.985 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:49.989 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:49.990 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:49.992 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2023-11-27 13:36:50.048 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20231128 2023-11-27 13:36:50.061 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:50.061 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:50.065 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20231128 values less than (1701100800)) 2023-11-27 13:36:50.105 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063410, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:50.112 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063410, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:50.112 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063410, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:50.124 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063410, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:50.131 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063410, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:50.909 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:50.910 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:50.911 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2023-11-27 13:36:51.108 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:36:51.110 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2023-11-27 13:36:51.123 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20231129 2023-11-27 13:36:51.127 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:51.127 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:51.128 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20231129 values less than (1701187200)) 2023-11-27 13:36:51.343 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:36:51.349 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:36:51.608 INFO 10 --- [pool-ocp-schedules-19,,5799345a1662] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-11-27 13:36:51.713 INFO 10 --- [pool-ocp-schedules-19,,5799345a1662] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-11-27 13:36:51.731 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:51.732 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:51.733 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2023-11-27 13:36:51.804 INFO 10 --- [pool-overview-container-1,,9570a1fedf9d] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:36:51.873 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2023-11-27 13:36:51.896 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20231130 2023-11-27 13:36:51.924 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:51.924 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:51.929 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20231130 values less than (1701273600)) 2023-11-27 13:36:52.021 INFO 10 --- [pool-scanner--1567-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=132 2023-11-27 13:36:52.098 INFO 10 --- [pool-ocp-schedules-17,,6921433af65a] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-11-27 13:36:52.104 INFO 10 --- [pool-ocp-schedules-4,,2f4e232c0e7f] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-11-27 13:36:52.105 INFO 10 --- [pool-ocp-schedules-11,,3a3b7af46695] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-11-27 13:36:52.133 INFO 10 --- [pool-scanner--1568-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=132 2023-11-27 13:36:52.147 INFO 10 --- [pool-ocp-schedules-17,,6921433af65a] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-11-27 13:36:52.180 INFO 10 --- [pool-ocp-schedules-4,,2f4e232c0e7f] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-11-27 13:36:52.182 INFO 10 --- [pool-ocp-schedules-11,,3a3b7af46695] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-11-27 13:36:52.256 INFO 10 --- [pool-scanner--1569-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=132 2023-11-27 13:36:52.453 INFO 10 --- [pool-scanner--1570-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=132 2023-11-27 13:36:52.454 INFO 10 --- [ForkJoinPool-18-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:52.455 INFO 10 --- [ForkJoinPool-18-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:52.456 INFO 10 --- [ForkJoinPool-18-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:52.460 INFO 10 --- [ForkJoinPool-18-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:52.466 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:52.470 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:52.470 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:52.476 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:52.480 INFO 10 --- [ForkJoinPool-18-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:52.480 INFO 10 --- [ForkJoinPool-18-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:52.482 INFO 10 --- [ForkJoinPool-18-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:52.484 INFO 10 --- [ForkJoinPool-18-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:52.490 INFO 10 --- [ForkJoinPool-18-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:52.491 INFO 10 --- [ForkJoinPool-18-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:52.491 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:52.491 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:52.498 INFO 10 --- [ForkJoinPool-18-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:52.500 INFO 10 --- [ForkJoinPool-18-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=3.1.4 2023-11-27 13:36:52.504 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, MAX(CASE `name` WHEN 'region' THEN `info` ELSE '' END ) `region`, MAX(CASE `name` WHEN 'idc' THEN `info` ELSE '' END ) `idc`, MAX(CASE `name` WHEN 'status' THEN `info` ELSE '' END ) `status`, MAX(CASE `name` WHEN 'merge_status' THEN `info` ELSE '' END ) `merge_status`, MAX(CASE `name` WHEN 'broadcast_version' THEN `value` ELSE 0 END ) `broadcast_version`, MAX(CASE `name` WHEN 'all_merged_version' THEN `value` ELSE 0 END ) `all_merged_version`, MAX(CASE `name` WHEN 'last_merged_version' THEN `value` ELSE 0 END ) `last_merged_version`, MAX(CASE `name` WHEN 'merge_start_time' THEN `value` ELSE 0 END ) `merge_start_time`, MAX(CASE `name` WHEN 'last_merged_time' THEN `value` ELSE 0 END ) `last_merged_time`, MAX(CASE `name` WHEN 'is_merge_timeout' THEN `value` ELSE 0 END ) `merge_timeout` FROM oceanbase.__all_zone WHERE `zone` <> '' GROUP BY `zone` 2023-11-27 13:36:52.507 INFO 10 --- [ForkJoinPool-18-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:52.507 INFO 10 --- [ForkJoinPool-18-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:52.514 INFO 10 --- [ForkJoinPool-18-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:52.520 INFO 10 --- [ForkJoinPool-18-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-11-27 13:36:52.549 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:52.550 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:52.554 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2023-11-27 13:36:52.596 INFO 10 --- [pool-ocp-schedules-6,,4392a9c4e895] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=4 2023-11-27 13:36:52.606 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:52.607 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:52.607 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:52.609 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:52.614 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:52.614 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:52.615 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-11-27 13:36:52.623 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2023-11-27 13:36:52.625 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=metric_data_minute, partition=p20231025 2023-11-27 13:36:52.637 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:52.637 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:52.645 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` drop partition (p20231025) 2023-11-27 13:36:52.710 INFO 10 --- [pool-ocp-schedules-15,,533c50764e95] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:36:52.781 INFO 10 --- [pool-ocp-schedules-15,,533c50764e95] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:36:52.882 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:52.882 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:52.883 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2023-11-27 13:36:52.939 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=metric_data_minute, partition=p20231026 2023-11-27 13:36:52.941 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:52.942 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:52.943 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` drop partition (p20231026) 2023-11-27 13:36:53.172 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:53.172 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:53.173 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2023-11-27 13:36:53.205 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=metric_data_minute, partition=p20231027 2023-11-27 13:36:53.206 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:53.206 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:53.206 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` drop partition (p20231027) 2023-11-27 13:36:53.270 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.120.0.1:8080 2023-11-27 13:36:53.375 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:36:53.372414+08:00, lastProcessTime=2023-11-27T13:36:43.129618+08:00 2023-11-27 13:36:53.423 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:53.424 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:53.425 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2023-11-27 13:36:53.482 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:53.482 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:53.501 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:53.504 INFO 10 --- [pool-ocp-schedules-15,,533c50764e95] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:36:53.516 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:53.517 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:53.523 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2023-11-27 13:36:53.583 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.120.0.1:8080 2023-11-27 13:36:53.606 INFO 10 --- [pool-ocp-schedules-15,,533c50764e95] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:36:53.655 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:53.656 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:53.662 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:53.675 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:53.675 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:53.684 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2023-11-27 13:36:53.721 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.120.0.1:8080 2023-11-27 13:36:53.744 INFO 10 --- [pool-ocp-schedules-14,,1518b32687b4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:36:53.744 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:53.746 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:53.755 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:53.756 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:53.759 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:53.775 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:53.775 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:53.780 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, MAX(CASE `name` WHEN 'region' THEN `info` ELSE '' END ) `region`, MAX(CASE `name` WHEN 'idc' THEN `info` ELSE '' END ) `idc`, MAX(CASE `name` WHEN 'status' THEN `info` ELSE '' END ) `status`, MAX(CASE `name` WHEN 'merge_status' THEN `info` ELSE '' END ) `merge_status`, MAX(CASE `name` WHEN 'broadcast_version' THEN `value` ELSE 0 END ) `broadcast_version`, MAX(CASE `name` WHEN 'all_merged_version' THEN `value` ELSE 0 END ) `all_merged_version`, MAX(CASE `name` WHEN 'last_merged_version' THEN `value` ELSE 0 END ) `last_merged_version`, MAX(CASE `name` WHEN 'merge_start_time' THEN `value` ELSE 0 END ) `merge_start_time`, MAX(CASE `name` WHEN 'last_merged_time' THEN `value` ELSE 0 END ) `last_merged_time`, MAX(CASE `name` WHEN 'is_merge_timeout' THEN `value` ELSE 0 END ) `merge_timeout` FROM oceanbase.__all_zone WHERE `zone` <> '' GROUP BY `zone` 2023-11-27 13:36:53.782 WARN 10 --- [ForkJoinPool-20-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-11-27 13:36:53.785 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:53.786 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:53.786 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, stop_time, start_service_time, block_migrate_in_time, id FROM oceanbase.__all_server ORDER BY zone ASC, id ASC 2023-11-27 13:36:53.787 INFO 10 --- [ForkJoinPool-20-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.206.97:2882, status=ACTIVE, startServiceTime=1701063018577428, active=true 2023-11-27 13:36:53.787 INFO 10 --- [ForkJoinPool-20-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=133.197.206.97:2882, stopTime=0, stopped=false 2023-11-27 13:36:53.787 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=IDLE, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=IDLE, startTime=null, stopTime=null} 2023-11-27 13:36:53.890 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=IDLE, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:36:53.890 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=133.197.206.97, port=2882, sqlPort=2881, version=3.1.4_10000092022071511-b4bfa011ceaef428782dcb65ae89190c40b78c2f(Jul 15 2022 11:45:14), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-27T13:30:19+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.206.97, port=2882, sqlPort=2881, version=3.1.4_10000092022071511-b4bfa011ceaef428782dcb65ae89190c40b78c2f(Jul 15 2022 11:45:14), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-27T05:30:18.577428Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:36:53.942 INFO 10 --- [pool-ocp-schedules-15,,533c50764e95] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:36:53.959 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:53.959 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:53.972 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:53.979 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=133.197.206.97, port=2882, sqlPort=2881, version=3.1.4_10000092022071511-b4bfa011ceaef428782dcb65ae89190c40b78c2f(Jul 15 2022 11:45:14), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-27T05:30:18.577428Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:36:53.997 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:53.997 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:54.025 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2023-11-27 13:36:54.079 INFO 10 --- [pool-ocp-schedules-15,,533c50764e95] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:36:54.091 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:36:53.372414+08:00 2023-11-27 13:36:54.138 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=sql_aggregate_data, partition=p1701100800 2023-11-27 13:36:54.141 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:54.141 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:54.146 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `sql_aggregate_data` add partition(partition p1701100800 values less than (1701100800)) 2023-11-27 13:36:54.219 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:54.224 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:54.227 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:54.228 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:54.230 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:54.236 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:54.237 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:54.237 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:36:54.239 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:54.314 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:54.323 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:54.325 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:54.326 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:54.327 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:54.341 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:54.341 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:54.350 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:36:54.351 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:54.355 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:36:54.368 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:54.368 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:54.369 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT cluster_id, cluster_name, created, cluster_role, cluster_status, current_scn, standby_became_primary_scn, primary_cluster_id, (time_to_usec(now(6)) - current_scn)/1000000 as delay_time, protection_mode, redo_transport_options, protection_level FROM oceanbase.v$ob_cluster 2023-11-27 13:36:54.382 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:36:54.383 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:54.388 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT switchover_status, switchover_info FROM oceanbase.v$ob_cluster 2023-11-27 13:36:54.426 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=obcluster, obClusterId=1 2023-11-27 13:36:54.426 INFO 10 --- [ForkJoinPool-20-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"133.197.206.97:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1700793672036000}, createTime=2022-11-29T14:52:53+08:00, creator=admin, obVersion=3.1.4, tenantCount=3) 2023-11-27 13:36:54.434 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.120.0.1:8080 2023-11-27 13:36:54.690 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:36:54.690 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:36:54.769 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:36:54.831 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:54.831 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:54.862 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='sql_aggregate_data' and table_type='BASE TABLE' 2023-11-27 13:36:54.965 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:54.967 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:54.976 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:54.976 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:54.980 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:54.984 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:54.984 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:54.987 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-11-27 13:36:55.065 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:55.065 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:55.067 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-11-27 13:36:55.070 INFO 10 --- [ForkJoinPool-20-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.204.3:2882, status=INACTIVE, startServiceTime=0, active=false 2023-11-27 13:36:55.070 INFO 10 --- [ForkJoinPool-20-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.204.10:2882, status=ACTIVE, startServiceTime=1695730092226557, active=true 2023-11-27 13:36:55.070 INFO 10 --- [ForkJoinPool-20-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=133.197.204.10:2882, stopTime=0, stopped=false 2023-11-27 13:36:55.070 INFO 10 --- [ForkJoinPool-20-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.204.11:2882, status=ACTIVE, startServiceTime=1695730305487530, active=true 2023-11-27 13:36:55.070 INFO 10 --- [ForkJoinPool-20-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=133.197.204.11:2882, stopTime=0, stopped=false 2023-11-27 13:36:55.070 INFO 10 --- [ForkJoinPool-20-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.204.12:2882, status=ACTIVE, startServiceTime=1695730736949667, active=true 2023-11-27 13:36:55.071 INFO 10 --- [ForkJoinPool-20-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=133.197.204.12:2882, stopTime=0, stopped=false 2023-11-27 13:36:55.071 INFO 10 --- [ForkJoinPool-20-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.204.13:2882, status=ACTIVE, startServiceTime=1695731005656813, active=true 2023-11-27 13:36:55.071 INFO 10 --- [ForkJoinPool-20-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=133.197.204.13:2882, stopTime=0, stopped=false 2023-11-27 13:36:55.071 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=14, name=zone6, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone6, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-27 13:36:55.272 INFO 10 --- [pool-prometheus-second-schedule-1,,08ea6c27f4c9] c.o.o.m.helper.ExporterRequestHelper : Get http client too slow, exporterAddr=OcpExporterAddr(id=2000134, exporterType=OB_PROXY, jobName=ob, hostId=4, instance=http://133.197.204.10:62889/metrics/obproxy, scrapeInterval=1, status=active, gmtCreate=2023-04-25 14:57:43.0, gmtModified=2023-11-27 13:36:42.0, uri=/obproxy, path=/metrics/obproxy, port=62889), start=1701063415107, elapsed=165 2023-11-27 13:36:55.273 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063415, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:55.275 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063415, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:55.277 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063415, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:55.286 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063415, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:55.296 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063415, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:36:55.318 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=sql_aggregate_data 2023-11-27 13:36:55.326 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=sql_aggregate_data, partition=p1701187200 2023-11-27 13:36:55.327 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:55.328 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:55.333 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `sql_aggregate_data` add partition(partition p1701187200 values less than (1701187200)) 2023-11-27 13:36:55.376 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=14, name=zone6, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:36:55.376 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=14, ip=133.197.204.13, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=7, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T20:23:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.204.13, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=7, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:23:25.656813Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:36:55.485 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=14, ip=133.197.204.13, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=7, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:23:25.656813Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:36:55.485 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=13, name=zone5, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone5, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-27 13:36:55.524 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:36:55.580 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=13, name=zone5, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:36:55.580 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=13, ip=133.197.204.12, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=6, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T20:18:57+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.204.12, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=6, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:18:56.949667Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:36:55.595 INFO 10 --- [pool-ocp-schedules-16,,270e0486fd7c] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:36:55.596 INFO 10 --- [pool-ocp-schedules-16,,270e0486fd7c] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:36:55.629 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=13, ip=133.197.204.12, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=6, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:18:56.949667Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:36:55.629 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=12, name=zone4, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone4, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-27 13:36:55.687 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:55.688 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:55.697 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='sql_aggregate_data' and table_type='BASE TABLE' 2023-11-27 13:36:55.747 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=12, name=zone4, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:36:55.747 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=12, ip=133.197.204.11, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T20:11:45+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.204.11, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:11:45.487530Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:36:55.770 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=sql_aggregate_data 2023-11-27 13:36:55.791 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=sql_aggregate_data, partition=p1701273600 2023-11-27 13:36:55.826 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:55.827 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:55.832 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `sql_aggregate_data` add partition(partition p1701273600 values less than (1701273600)) 2023-11-27 13:36:55.851 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:36:55.900 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=12, ip=133.197.204.11, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:11:45.487530Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:36:55.900 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=11, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-27 13:36:56.080 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=11, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:36:56.080 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=11, ip=133.197.204.10, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=true, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T20:08:12+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.204.10, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=true, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:08:12.226557Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:36:56.085 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:36:56.575 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:56.575 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:56.587 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='sql_aggregate_data' and table_type='BASE TABLE' 2023-11-27 13:36:56.602 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=11, ip=133.197.204.10, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=true, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:08:12.226557Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:36:56.602 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-27 13:36:56.673 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=sql_aggregate_data 2023-11-27 13:36:56.681 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:36:56.681 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=133.197.204.3, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=2, status=UNAVAILABLE, innerStatus=INACTIVE, startTime=1970-01-01T08:00+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.204.3, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=2, status=null, innerStatus=INACTIVE, startTime=1970-01-01T00:00Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:36:56.683 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=sql_aggregate_data, partition=p1700150400 2023-11-27 13:36:56.697 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:56.697 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:56.708 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `sql_aggregate_data` drop partition (p1700150400) 2023-11-27 13:36:56.818 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=133.197.204.3, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=2, status=UNAVAILABLE, innerStatus=INACTIVE, startTime=1970-01-01T00:00Z, stopTime=1970-01-01T00:00Z}, oldStatus=UNAVAILABLE, newStatus=UNAVAILABLE 2023-11-27 13:36:56.945 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:36:56.956 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:56.963 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:56.964 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:56.970 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:56.982 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:56.982 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:56.982 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:36:56.983 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:56.988 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='sql_aggregate_data' and table_type='BASE TABLE' 2023-11-27 13:36:56.988 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:36:57.027 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=sql_aggregate_data, partition=p1700236800 2023-11-27 13:36:57.035 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:57.036 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:57.044 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `sql_aggregate_data` drop partition (p1700236800) 2023-11-27 13:36:57.188 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=darenOB, obClusterId=10 2023-11-27 13:36:57.188 INFO 10 --- [ForkJoinPool-20-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=darenOB, obClusterId=10, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"darenOB","ObClusterId":10,"ObRegion":"darenOB","ObRegionId":10,"ReadonlyRsList":[],"RsList":[{"address":"133.197.204.10:2882","role":"LEADER","sql_port":2881},{"address":"133.197.204.11:2882","role":"FOLLOWER","sql_port":2881},{"address":"133.197.204.12:2882","role":"FOLLOWER","sql_port":2881},{"address":"133.197.204.13:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1701063392653000}, createTime=2022-11-29T15:05:37+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-11-27 13:36:57.213 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:57.214 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:57.224 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='sql_aggregate_data' and table_type='BASE TABLE' 2023-11-27 13:36:57.245 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=sql_aggregate_data, partition=p1700323200 2023-11-27 13:36:57.252 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:57.253 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:57.259 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `sql_aggregate_data` drop partition (p1700323200) 2023-11-27 13:36:57.356 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:36:57.362 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:36:57.389 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:57.390 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:57.397 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='sql_aggregate_data' and table_type='BASE TABLE' 2023-11-27 13:36:57.429 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:57.430 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:57.437 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:57.457 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:57.458 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:57.466 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2023-11-27 13:36:57.524 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:57.525 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:57.536 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:36:57.551 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:57.552 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:57.564 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2023-11-27 13:36:57.589 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_tenant_mini_compaction_info, partition=p1701100800 2023-11-27 13:36:57.592 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:57.592 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:57.609 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_tenant_mini_compaction_info` add partition(partition p1701100800 values less than (1701100800)) 2023-11-27 13:36:58.159 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:58.159 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:58.163 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_tenant_mini_compaction_info' and table_type='BASE TABLE' 2023-11-27 13:36:58.321 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_tenant_mini_compaction_info 2023-11-27 13:36:58.323 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_tenant_mini_compaction_info, partition=p1701187200 2023-11-27 13:36:58.324 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:58.324 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:58.324 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_tenant_mini_compaction_info` add partition(partition p1701187200 values less than (1701187200)) 2023-11-27 13:36:58.613 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:58.613 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:58.617 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_tenant_mini_compaction_info' and table_type='BASE TABLE' 2023-11-27 13:36:58.751 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_tenant_mini_compaction_info 2023-11-27 13:36:58.758 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_tenant_mini_compaction_info, partition=p1701273600 2023-11-27 13:36:58.768 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:58.768 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:58.779 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_tenant_mini_compaction_info` add partition(partition p1701273600 values less than (1701273600)) 2023-11-27 13:36:59.202 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:59.202 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:59.208 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_tenant_mini_compaction_info' and table_type='BASE TABLE' 2023-11-27 13:36:59.288 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_tenant_mini_compaction_info 2023-11-27 13:36:59.299 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_tenant_mini_compaction_info, partition=p1700236800 2023-11-27 13:36:59.301 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:59.301 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:59.302 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_tenant_mini_compaction_info` drop partition (p1700236800) 2023-11-27 13:36:59.503 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:59.503 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:59.508 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_tenant_mini_compaction_info' and table_type='BASE TABLE' 2023-11-27 13:36:59.524 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_tenant_mini_compaction_info, partition=p1700323200 2023-11-27 13:36:59.526 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:59.527 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:59.533 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_tenant_mini_compaction_info` drop partition (p1700323200) 2023-11-27 13:36:59.570 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:36:59.840 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:59.840 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:59.841 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_tenant_mini_compaction_info' and table_type='BASE TABLE' 2023-11-27 13:36:59.856 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Monitordb, table=ob_tenant_mini_compaction_info, partition=p1700409600 2023-11-27 13:36:59.857 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:59.857 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:59.858 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_tenant_mini_compaction_info` drop partition (p1700409600) 2023-11-27 13:36:59.969 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:36:59.970 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:36:59.971 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_tenant_mini_compaction_info' and table_type='BASE TABLE' 2023-11-27 13:36:59.981 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2023-11-27 13:36:59.981 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2023-11-27 13:36:59.981 INFO 10 --- [part-initial-worker,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=proxy, clusterName=obcluster, obClusterId=null, tenantName=meta_tenant, username=meta_user, address=133.197.206.97, port=2883, database=meta_database) 2023-11-27 13:36:59.981 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=proxy, address=133.197.206.97, port=2883, obsAddrList=null, username=meta_user, tenantName=meta_tenant, clusterName=obcluster, obClusterId=null, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=meta_database) 2023-11-27 13:36:59.981 ERROR 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.monitor.query.MonitorServiceImpl : get metric data metric:system_event_delta_total_waits, labelMap:{app=ob, obregion=darenOB, ob_cluster_id=10}, prometheusLabels:[], gbLabels:[app, obregion, ob_cluster_id, ob_tenant_id, event_id, event_name, wait_class]. ex: {} java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: org.springframework.dao.TransientDataAccessResourceException: PreparedStatementCallback; SQL [SELECT /*+ READ_CONSISTENCY(WEAK) */ `series_id`, `timestamp`, `data`, `interval` FROM `metric_data_minute` WHERE series_id IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) and timestamp >= ? and timestamp <= ? ORDER BY series_id]; (conn=1048714) Timeout; nested exception is java.sql.SQLTransientConnectionException: (conn=1048714) Timeout at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_312] at java.util.concurrent.FutureTask.get(FutureTask.java:206) ~[na:1.8.0_312] at com.oceanbase.ocp.monitor.query.MonitorServiceImpl.queryMetricBatch(MonitorServiceImpl.java:134) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.query.MetricServiceImpl.getAuthorizedQueryMap(MetricServiceImpl.java:173) [ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.query.MetricServiceImpl.queryMetricsWithLabel(MetricServiceImpl.java:104) [ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.perf.awr.internal.operator.SystemEventOperator.innerGenerateSnapshot(SystemEventOperator.java:82) [ocp-perf-awr-pipeline-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.perf.awr.internal.operator.SystemEventOperator.generateSnapshot(SystemEventOperator.java:72) [ocp-perf-awr-pipeline-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.perf.sql.diag.internal.GenerateSnapshotService.generateSnapshotData(GenerateSnapshotService.java:163) [ocp-perf-sql-diag-pipeline-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.perf.sql.diag.internal.task.GenerateSnapshotTask.run(GenerateSnapshotTask.java:85) [ocp-perf-sql-diag-pipeline-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:60) [ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) [ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) [ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:111) [ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.redirectOutputIfNotSysSchedule(RunnerFactory.java:174) [ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.run(RunnerFactory.java:101) [ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.ReadySubtaskWorker.lambda$null$1(ReadySubtaskWorker.java:127) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.util.concurrent.ExecutionException: org.springframework.dao.TransientDataAccessResourceException: PreparedStatementCallback; SQL [SELECT /*+ READ_CONSISTENCY(WEAK) */ `series_id`, `timestamp`, `data`, `interval` FROM `metric_data_minute` WHERE series_id IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) and timestamp >= ? and timestamp <= ? ORDER BY series_id]; (conn=1048714) Timeout; nested exception is java.sql.SQLTransientConnectionException: (conn=1048714) Timeout at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_312] at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_312] at com.oceanbase.ocp.monitor.calculate.OcpPrometheusAstNode.doGet(OcpPrometheusAstNode.java:173) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.calculate.OcpPrometheusAstNode.afterPrepare(OcpPrometheusAstNode.java:153) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.calculate.OcpPrometheusAstNode.afterPrepare(OcpPrometheusAstNode.java:146) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.calculate.OcpPrometheusAstNode.afterPrepare(OcpPrometheusAstNode.java:146) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.calculate.OcpPrometheusAstNode.eval(OcpPrometheusAstNode.java:231) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.query.MonitorServiceImpl.doQuery(MonitorServiceImpl.java:336) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.query.MonitorServiceImpl.lambda$queryMetricBatch$0(MonitorServiceImpl.java:122) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] ... 4 common frames omitted Caused by: org.springframework.dao.TransientDataAccessResourceException: PreparedStatementCallback; SQL [SELECT /*+ READ_CONSISTENCY(WEAK) */ `series_id`, `timestamp`, `data`, `interval` FROM `metric_data_minute` WHERE series_id IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) and timestamp >= ? and timestamp <= ? ORDER BY series_id]; (conn=1048714) Timeout; nested exception is java.sql.SQLTransientConnectionException: (conn=1048714) Timeout at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:70) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:70) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:79) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1541) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:667) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:713) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:738) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:794) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.query(NamedParameterJdbcTemplate.java:209) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at com.oceanbase.ocp.monitor.storage.RollupMetricDataDao.mapToValueNodes(RollupMetricDataDao.java:116) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.storage.RollupMetricDataDao.scan(RollupMetricDataDao.java:111) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.storage.RollupMetricDataDaoProxy.scan(RollupMetricDataDaoProxy.java:86) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.store.RollupMetricDataStore.fetchNodeFromDb(RollupMetricDataStore.java:188) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.store.RollupMetricDataStore.ranges(RollupMetricDataStore.java:110) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.store.RollupMetricDataStore.ranges(RollupMetricDataStore.java:94) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.store.MetricDataStoreFactory.ranges(MetricDataStoreFactory.java:50) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.service.OcpPrometheusScanBusinessImpl.scan(OcpPrometheusScanBusinessImpl.java:90) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.query.util.MetricQueryScanner.lambda$scan$0(MetricQueryScanner.java:93) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] ... 4 common frames omitted Caused by: java.sql.SQLTransientConnectionException: (conn=1048714) Timeout at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:202) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseStatement.executeExceptionEpilogue(OceanBaseStatement.java:315) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:243) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.JDBC4PreparedStatement.execute(JDBC4PreparedStatement.java:161) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.JDBC4PreparedStatement.executeQuery(JDBC4PreparedStatement.java:175) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidPooledPreparedStatement.executeQuery(DruidPooledPreparedStatement.java:227) ~[druid-1.2.8.jar!/:1.2.8] at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:722) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:651) ~[spring-jdbc-5.3.23.jar!/:5.3.23] ... 17 common frames omitted Caused by: com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException: Timeout at com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException.of(OceanBaseSqlException.java:79) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:199) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:180) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:372) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:234) ~[oceanbase-client-2.4.2.jar!/:na] ... 22 common frames omitted Caused by: java.sql.SQLException: Timeout at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:2192) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2057) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:1951) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:370) ~[oceanbase-client-2.4.2.jar!/:na] ... 23 common frames omitted 2023-11-27 13:37:00.001 INFO 10 --- [pool-ocp-schedules-2,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-11-27 13:37:00.001 INFO 10 --- [pool-ocp-schedules-2,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-11-27 13:37:00.001 INFO 10 --- [pool-ocp-schedules-6,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-11-27 13:37:00.001 INFO 10 --- [pool-ocp-schedules-6,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1701063420 2023-11-27 13:37:00.058 INFO 10 --- [obsdk-init-thread-4,,0819d06e39b7] com.alibaba.druid.pool.DruidDataSource : {dataSource-7,ob-connector-datasource_proxy_obcluster_null_meta_tenant_meta_user_133.197.206.97_2883} inited 2023-11-27 13:37:00.062 INFO 10 --- [obsdk-init-thread-4,,0819d06e39b7] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource success, poolingCount=1, connectProperties=ConnectProperties(connectionMode=proxy, address=133.197.206.97, port=2883, obsAddrList=null, username=meta_user, tenantName=meta_tenant, clusterName=obcluster, obClusterId=null, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=meta_database) 2023-11-27 13:37:00.064 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector success 2023-11-27 13:37:00.064 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:00.069 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:00.069 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:00.073 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2023-11-27 13:37:00.103 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063420, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:00.113 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063420, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:00.114 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063420, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:00.126 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063420, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:00.181 INFO 10 --- [http-nio-0.0.0.0-8080-exec-2,e4b8c2e382ff470d,ba25c00595d6] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.11, traceId=e4b8c2e382ff470d, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:37:00.746 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063420, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:00.748 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:37:00.749 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:37:00.763 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:00.764 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:00.793 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:37:00.796 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:00.796 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:00.797 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2023-11-27 13:37:00.800 ERROR 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.monitor.query.MonitorServiceImpl : get metric data metric:system_event_delta_event_total_timeouts, labelMap:{app=ob, obregion=darenOB, ob_cluster_id=10}, prometheusLabels:[], gbLabels:[app, obregion, ob_cluster_id, ob_tenant_id, event_id, event_name, wait_class]. ex: {} java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: org.springframework.dao.TransientDataAccessResourceException: PreparedStatementCallback; SQL [SELECT /*+ READ_CONSISTENCY(WEAK) */ `series_id`, `timestamp`, `data`, `interval` FROM `metric_data_minute` WHERE series_id IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) and timestamp >= ? and timestamp <= ? ORDER BY series_id]; (conn=196) Timeout; nested exception is java.sql.SQLTransientConnectionException: (conn=196) Timeout at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_312] at java.util.concurrent.FutureTask.get(FutureTask.java:206) ~[na:1.8.0_312] at com.oceanbase.ocp.monitor.query.MonitorServiceImpl.queryMetricBatch(MonitorServiceImpl.java:134) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.query.MetricServiceImpl.getAuthorizedQueryMap(MetricServiceImpl.java:173) [ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.query.MetricServiceImpl.queryMetricsWithLabel(MetricServiceImpl.java:104) [ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.perf.awr.internal.operator.SystemEventOperator.innerGenerateSnapshot(SystemEventOperator.java:82) [ocp-perf-awr-pipeline-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.perf.awr.internal.operator.SystemEventOperator.generateSnapshot(SystemEventOperator.java:72) [ocp-perf-awr-pipeline-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.perf.sql.diag.internal.GenerateSnapshotService.generateSnapshotData(GenerateSnapshotService.java:163) [ocp-perf-sql-diag-pipeline-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.perf.sql.diag.internal.task.GenerateSnapshotTask.run(GenerateSnapshotTask.java:85) [ocp-perf-sql-diag-pipeline-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:60) [ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) [ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) [ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:111) [ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.redirectOutputIfNotSysSchedule(RunnerFactory.java:174) [ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.run(RunnerFactory.java:101) [ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.ReadySubtaskWorker.lambda$null$1(ReadySubtaskWorker.java:127) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.util.concurrent.ExecutionException: org.springframework.dao.TransientDataAccessResourceException: PreparedStatementCallback; SQL [SELECT /*+ READ_CONSISTENCY(WEAK) */ `series_id`, `timestamp`, `data`, `interval` FROM `metric_data_minute` WHERE series_id IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) and timestamp >= ? and timestamp <= ? ORDER BY series_id]; (conn=196) Timeout; nested exception is java.sql.SQLTransientConnectionException: (conn=196) Timeout at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_312] at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_312] at com.oceanbase.ocp.monitor.calculate.OcpPrometheusAstNode.doGet(OcpPrometheusAstNode.java:173) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.calculate.OcpPrometheusAstNode.afterPrepare(OcpPrometheusAstNode.java:153) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.calculate.OcpPrometheusAstNode.afterPrepare(OcpPrometheusAstNode.java:146) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.calculate.OcpPrometheusAstNode.afterPrepare(OcpPrometheusAstNode.java:146) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.calculate.OcpPrometheusAstNode.eval(OcpPrometheusAstNode.java:231) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.query.MonitorServiceImpl.doQuery(MonitorServiceImpl.java:336) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.query.MonitorServiceImpl.lambda$queryMetricBatch$0(MonitorServiceImpl.java:122) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] ... 4 common frames omitted Caused by: org.springframework.dao.TransientDataAccessResourceException: PreparedStatementCallback; SQL [SELECT /*+ READ_CONSISTENCY(WEAK) */ `series_id`, `timestamp`, `data`, `interval` FROM `metric_data_minute` WHERE series_id IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) and timestamp >= ? and timestamp <= ? ORDER BY series_id]; (conn=196) Timeout; nested exception is java.sql.SQLTransientConnectionException: (conn=196) Timeout at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:70) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:70) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:79) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1541) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:667) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:713) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:738) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:794) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.query(NamedParameterJdbcTemplate.java:209) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at com.oceanbase.ocp.monitor.storage.RollupMetricDataDao.mapToValueNodes(RollupMetricDataDao.java:116) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.storage.RollupMetricDataDao.scan(RollupMetricDataDao.java:111) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.storage.RollupMetricDataDaoProxy.scan(RollupMetricDataDaoProxy.java:86) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.store.RollupMetricDataStore.fetchNodeFromDb(RollupMetricDataStore.java:188) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.store.RollupMetricDataStore.ranges(RollupMetricDataStore.java:110) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.store.RollupMetricDataStore.ranges(RollupMetricDataStore.java:94) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.store.MetricDataStoreFactory.ranges(MetricDataStoreFactory.java:50) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.service.OcpPrometheusScanBusinessImpl.scan(OcpPrometheusScanBusinessImpl.java:90) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.query.util.MetricQueryScanner.lambda$scan$0(MetricQueryScanner.java:93) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] ... 4 common frames omitted Caused by: java.sql.SQLTransientConnectionException: (conn=196) Timeout at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:202) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseStatement.executeExceptionEpilogue(OceanBaseStatement.java:315) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:243) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.JDBC4PreparedStatement.execute(JDBC4PreparedStatement.java:161) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.JDBC4PreparedStatement.executeQuery(JDBC4PreparedStatement.java:175) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidPooledPreparedStatement.executeQuery(DruidPooledPreparedStatement.java:227) ~[druid-1.2.8.jar!/:1.2.8] at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:722) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:651) ~[spring-jdbc-5.3.23.jar!/:5.3.23] ... 17 common frames omitted Caused by: com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException: Timeout at com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException.of(OceanBaseSqlException.java:79) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:199) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:180) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:372) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:234) ~[oceanbase-client-2.4.2.jar!/:na] ... 22 common frames omitted Caused by: java.sql.SQLException: Timeout at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:2192) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2057) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:1951) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:370) ~[oceanbase-client-2.4.2.jar!/:na] ... 23 common frames omitted 2023-11-27 13:37:00.958 ERROR 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.monitor.query.MonitorServiceImpl : get metric data metric:system_event_delta_event_time_waited_us, labelMap:{app=ob, obregion=darenOB, ob_cluster_id=10}, prometheusLabels:[], gbLabels:[app, obregion, ob_cluster_id, ob_tenant_id, event_id, event_name, wait_class]. ex: {} java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: org.springframework.dao.TransientDataAccessResourceException: PreparedStatementCallback; SQL [SELECT /*+ READ_CONSISTENCY(WEAK) */ `series_id`, `timestamp`, `data`, `interval` FROM `metric_data_minute` WHERE series_id IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) and timestamp >= ? and timestamp <= ? ORDER BY series_id]; (conn=183) Timeout; nested exception is java.sql.SQLTransientConnectionException: (conn=183) Timeout at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_312] at java.util.concurrent.FutureTask.get(FutureTask.java:206) ~[na:1.8.0_312] at com.oceanbase.ocp.monitor.query.MonitorServiceImpl.queryMetricBatch(MonitorServiceImpl.java:134) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.query.MetricServiceImpl.getAuthorizedQueryMap(MetricServiceImpl.java:173) [ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.query.MetricServiceImpl.queryMetricsWithLabel(MetricServiceImpl.java:104) [ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.perf.awr.internal.operator.SystemEventOperator.innerGenerateSnapshot(SystemEventOperator.java:82) [ocp-perf-awr-pipeline-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.perf.awr.internal.operator.SystemEventOperator.generateSnapshot(SystemEventOperator.java:72) [ocp-perf-awr-pipeline-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.perf.sql.diag.internal.GenerateSnapshotService.generateSnapshotData(GenerateSnapshotService.java:163) [ocp-perf-sql-diag-pipeline-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.perf.sql.diag.internal.task.GenerateSnapshotTask.run(GenerateSnapshotTask.java:85) [ocp-perf-sql-diag-pipeline-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:60) [ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) [ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) [ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:111) [ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.redirectOutputIfNotSysSchedule(RunnerFactory.java:174) [ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.run(RunnerFactory.java:101) [ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.ReadySubtaskWorker.lambda$null$1(ReadySubtaskWorker.java:127) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.util.concurrent.ExecutionException: org.springframework.dao.TransientDataAccessResourceException: PreparedStatementCallback; SQL [SELECT /*+ READ_CONSISTENCY(WEAK) */ `series_id`, `timestamp`, `data`, `interval` FROM `metric_data_minute` WHERE series_id IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) and timestamp >= ? and timestamp <= ? ORDER BY series_id]; (conn=183) Timeout; nested exception is java.sql.SQLTransientConnectionException: (conn=183) Timeout at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_312] at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_312] at com.oceanbase.ocp.monitor.calculate.OcpPrometheusAstNode.doGet(OcpPrometheusAstNode.java:173) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.calculate.OcpPrometheusAstNode.afterPrepare(OcpPrometheusAstNode.java:153) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.calculate.OcpPrometheusAstNode.afterPrepare(OcpPrometheusAstNode.java:146) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.calculate.OcpPrometheusAstNode.afterPrepare(OcpPrometheusAstNode.java:146) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.calculate.OcpPrometheusAstNode.eval(OcpPrometheusAstNode.java:231) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.query.MonitorServiceImpl.doQuery(MonitorServiceImpl.java:336) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.query.MonitorServiceImpl.lambda$queryMetricBatch$0(MonitorServiceImpl.java:122) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] ... 4 common frames omitted Caused by: org.springframework.dao.TransientDataAccessResourceException: PreparedStatementCallback; SQL [SELECT /*+ READ_CONSISTENCY(WEAK) */ `series_id`, `timestamp`, `data`, `interval` FROM `metric_data_minute` WHERE series_id IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) and timestamp >= ? and timestamp <= ? ORDER BY series_id]; (conn=183) Timeout; nested exception is java.sql.SQLTransientConnectionException: (conn=183) Timeout at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:70) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:70) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:79) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1541) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:667) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:713) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:738) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:794) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.query(NamedParameterJdbcTemplate.java:209) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at com.oceanbase.ocp.monitor.storage.RollupMetricDataDao.mapToValueNodes(RollupMetricDataDao.java:116) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.storage.RollupMetricDataDao.scan(RollupMetricDataDao.java:111) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.storage.RollupMetricDataDaoProxy.scan(RollupMetricDataDaoProxy.java:86) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.store.RollupMetricDataStore.fetchNodeFromDb(RollupMetricDataStore.java:188) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.store.RollupMetricDataStore.ranges(RollupMetricDataStore.java:110) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.store.RollupMetricDataStore.ranges(RollupMetricDataStore.java:94) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.store.MetricDataStoreFactory.ranges(MetricDataStoreFactory.java:50) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.service.OcpPrometheusScanBusinessImpl.scan(OcpPrometheusScanBusinessImpl.java:90) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.monitor.query.util.MetricQueryScanner.lambda$scan$0(MetricQueryScanner.java:93) ~[ocp-monitor-service-4.0.3-20230301.jar!/:4.0.3-20230301] ... 4 common frames omitted Caused by: java.sql.SQLTransientConnectionException: (conn=183) Timeout at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:202) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseStatement.executeExceptionEpilogue(OceanBaseStatement.java:315) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:243) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.JDBC4PreparedStatement.execute(JDBC4PreparedStatement.java:161) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.JDBC4PreparedStatement.executeQuery(JDBC4PreparedStatement.java:175) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidPooledPreparedStatement.executeQuery(DruidPooledPreparedStatement.java:227) ~[druid-1.2.8.jar!/:1.2.8] at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:722) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:651) ~[spring-jdbc-5.3.23.jar!/:5.3.23] ... 17 common frames omitted Caused by: com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException: Timeout at com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException.of(OceanBaseSqlException.java:79) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:199) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:180) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:372) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:234) ~[oceanbase-client-2.4.2.jar!/:na] ... 22 common frames omitted Caused by: java.sql.SQLException: Timeout at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:2192) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2057) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:1951) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:370) ~[oceanbase-client-2.4.2.jar!/:na] ... 23 common frames omitted 2023-11-27 13:37:00.960 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.a.i.o.SystemEventOperator : [System event] store 0 targets 2023-11-27 13:37:00.960 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for SystemEventSnapshotOperator, error count is 0 2023-11-27 13:37:00.960 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate all snapshot, failCount = 0, beans = [sqlAuditSnapshotOperator, CacheSnapshotOperator, LatchSnapshotOperator, ParameterSnapshotOperator, RowLockSnapshotOperator, ClusterInfoSnapshotOperator, TenantInfoSnapshotOperator, ReplicaInfoOperator, SysStatSnapshotOperator, ServerInfoSnapshotOperator, PartitionHitOperator, RpcSnapshotOperator, MemoryOperator, SystemEventSnapshotOperator], fail beans is None, fail info is Node 2023-11-27 13:37:00.970 INFO 10 --- [pool-prometheus-minute-schedule-1,,1638f8532a4b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1701063480, interval=60 2023-11-27 13:37:00.974 INFO 10 --- [pool-prometheus-minute-schedule-2,,83a513d1ad37] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1701063480, interval=60 2023-11-27 13:37:01.010 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/extra, collectAt=1701063420, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:01.010 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62888/metrics/stat, collectAt=1701063420, message=java.net.ConnectException: Connection refused: /133.197.206.97:62888, rootCause=ConnectException: Connection refused 2023-11-27 13:37:01.010 INFO 10 --- [pool-prometheus-minute-schedule-1,,1638f8532a4b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=11, collectTaskCount=11 2023-11-27 13:37:01.011 INFO 10 --- [pool-prometheus-minute-schedule-1,,1638f8532a4b] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1701063420, count=11, elapsed=1699362357591 2023-11-27 13:37:01.011 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62888/metrics/stat, collectAt=1701063420, message=java.net.ConnectException: Connection refused: /133.197.206.88:62888, rootCause=ConnectException: Connection refused 2023-11-27 13:37:01.013 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/stat, collectAt=1701063420, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:01.062 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/stat, collectAt=1701063420, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:01.063 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/stat, collectAt=1701063420, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:01.065 INFO 10 --- [pool-prometheus-minute-schedule-2,,83a513d1ad37] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2023-11-27 13:37:01.065 INFO 10 --- [pool-prometheus-minute-schedule-2,,83a513d1ad37] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1701063420, count=8, elapsed=1699362357645 2023-11-27 13:37:01.068 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62888/metrics/stat, collectAt=1701063420, message=java.net.ConnectException: Connection refused: /133.197.206.92:62888, rootCause=ConnectException: Connection refused 2023-11-27 13:37:01.097 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:36:10.0, nextRun=2023-11-27 13:37:00.0, currentDate=Mon Nov 27 13:37:01 CST 2023 2023-11-27 13:37:01.097 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-27 13:36:10.0, nextRunTime=2023-11-27 13:37:00.0} 2023-11-27 13:37:01.124 INFO 10 --- [http-nio-0.0.0.0-8080-exec-2,e4b8c2e382ff470d,ba25c00595d6] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.11, traceId=e4b8c2e382ff470d, duration=944 ms] 2023-11-27 13:37:01.137 INFO 10 --- [http-nio-0.0.0.0-8080-exec-3,67704ba6150a4df8,293341772564] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.10, traceId=67704ba6150a4df8, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:37:01.172 INFO 10 --- [http-nio-0.0.0.0-8080-exec-3,67704ba6150a4df8,293341772564] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.10, traceId=67704ba6150a4df8, duration=36 ms] 2023-11-27 13:37:01.231 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c8cc17639a65] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000007, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:37:01.233 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=profile_credential_sharing, partition=p1701100800 2023-11-27 13:37:01.238 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:01.239 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:01.239 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `profile_credential_sharing` add partition(partition p1701100800 values less than (1701100800)) 2023-11-27 13:37:01.485 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-11-27 13:37:01.486 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:37:01.486 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-11-27 13:37:01.585 INFO 10 --- [pool-ocp-schedules-2,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-11-27 13:37:02.096 INFO 10 --- [pool-overview-container-1,,bfed07383661] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:37:02.148 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:36:10.0, nextRun=2023-11-27 13:37:00.0, currentDate=Mon Nov 27 13:37:02 CST 2023 2023-11-27 13:37:02.149 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-27 13:36:10.0, nextRunTime=2023-11-27 13:37:00.0} 2023-11-27 13:37:02.231 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-11-27 13:37:02.232 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:37:02.232 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-11-27 13:37:02.242 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000005, status:SUCCESS 2023-11-27 13:37:02.245 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000005, status:SUCCESS 2023-11-27 13:37:02.247 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:36:10.0, nextRun=2023-11-27 13:37:00.0, currentDate=Mon Nov 27 13:37:02 CST 2023 2023-11-27 13:37:02.247 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-27 13:36:10.0, nextRunTime=2023-11-27 13:37:00.0} 2023-11-27 13:37:02.271 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-11-27 13:37:02.271 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:37:02.271 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-11-27 13:37:02.307 INFO 10 --- [pool-scanner--1663-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=120 2023-11-27 13:37:02.311 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:36:34.0, nextRun=2023-11-27 13:37:00.0, currentDate=Mon Nov 27 13:37:02 CST 2023 2023-11-27 13:37:02.312 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-11-27 13:36:34.0, nextRunTime=2023-11-27 13:37:00.0} 2023-11-27 13:37:02.490 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-11-27 13:37:02.490 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:37:02.490 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-11-27 13:37:02.515 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:36:11.0, nextRun=2023-11-27 13:37:00.0, currentDate=Mon Nov 27 13:37:02 CST 2023 2023-11-27 13:37:02.515 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=1000002, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-27 13:36:11.0, nextRunTime=2023-11-27 13:37:00.0} 2023-11-27 13:37:02.538 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-11-27 13:37:02.538 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:37:02.538 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=1000002, state=RUNNING, operation=EXECUTE} 2023-11-27 13:37:03.216 INFO 10 --- [http-nio-0.0.0.0-8080-exec-8,badcdb9d8ef141ca,be1b320a530f] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.10, traceId=badcdb9d8ef141ca, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:37:03.225 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:37:03.220547+08:00, lastProcessTime=2023-11-27T13:36:53.372414+08:00 2023-11-27 13:37:03.263 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] slow query, durationMillis=2024, sql=alter table `profile_credential_sharing` add partition(partition p1701100800 values less than (1701100800)) 2023-11-27 13:37:03.265 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000040, operation:EXECUTE, state: RUNNING 2023-11-27 13:37:03.266 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:03.266 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:03.267 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='profile_credential_sharing' and table_type='BASE TABLE' 2023-11-27 13:37:03.275 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000040, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-11-27T13:37:03.225+08:00, endTime=null} 2023-11-27 13:37:03.297 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000041, operation:EXECUTE, state: RUNNING 2023-11-27 13:37:03.415 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000041, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-11-27T13:37:03.291+08:00, endTime=null} 2023-11-27 13:37:03.440 INFO 10 --- [http-nio-0.0.0.0-8080-exec-8,badcdb9d8ef141ca,be1b320a530f] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.10, traceId=badcdb9d8ef141ca, duration=224 ms] 2023-11-27 13:37:03.485 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,d41e5fe7ff9e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000040, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000033, task_operation=execute, latest_execution_start_time=2023-11-27T13:37:03.224+08:00, sub_task_instance_id=10000040}, listMap={}}, executor=10.120.0.1 2023-11-27 13:37:03.485 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,d41e5fe7ff9e] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-11-27 13:37:03.490 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000037, operation:EXECUTE, state: RUNNING 2023-11-27 13:37:03.505 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000037, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-11-27T13:37:03.486+08:00, endTime=null} 2023-11-27 13:37:03.513 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,51f3901dbc45] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000041, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000034, task_operation=execute, latest_execution_start_time=2023-11-27T13:37:03.290+08:00, sub_task_instance_id=10000041}, listMap={}}, executor=10.120.0.1 2023-11-27 13:37:03.547 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000038, operation:EXECUTE, state: RUNNING 2023-11-27 13:37:03.564 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:37:03.220547+08:00 2023-11-27 13:37:03.565 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000038, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-11-27T13:37:03.539+08:00, endTime=null} 2023-11-27 13:37:03.567 INFO 10 --- [pool-281-thread-2,554481c3fd1b426b,593ca3e2f4b2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:darenOB, tenant:daren 2023-11-27 13:37:03.567 INFO 10 --- [pool-281-thread-1,554481c3fd1b426b,74bc4170fc32] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:darenOB, tenant:sys 2023-11-27 13:37:03.574 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,21f56e121b00] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000037, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000030, task_operation=execute, latest_execution_start_time=2023-11-27T13:37:03.486+08:00, sub_task_instance_id=10000037}, listMap={}}, executor=10.120.0.1 2023-11-27 13:37:03.589 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,51f3901dbc45] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000041, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:37:03.630 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000039, operation:EXECUTE, state: RUNNING 2023-11-27 13:37:03.669 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000039, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-11-27T13:37:03.618+08:00, endTime=null} 2023-11-27 13:37:03.669 INFO 10 --- [pool-281-thread-1,554481c3fd1b426b,74bc4170fc32] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:03.673 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm is revoled, alarmId=19563, status=Inactive, labels=class OcpLabels { {app=OCP, alarm_name=分區創建失敗, app_type=OCP, alarm_name_en_us=partition create failed, alarm_name_zh_cn=分区创建失败, alarm_level=3, alarm_template_id=0, table_name=profile_credential_sharing, alarm_summary_zh_tw=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 分區創建失敗, result=success, alarm_summary_en_us=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing partition create failed, alarm_description_en_us=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing partition create failed partition (p1701100800) create failed, result success., alarm_description_zh_cn=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 分区创建失败 分区（p1701100800）创建失败，结果：success。, alarm_type=partition_create_failed, datasource=Metadb, alarm_status=Inactive, alarm_summary_zh_cn=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 分区创建失败, alarm_scope=Service, alarm_target=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing, ocp_ip=10.120.0.1, alarm_description_zh_tw=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 分區創建失敗 分區（p1701100800）創建失敗，結果：success。, partition_name=p1701100800, alarm_evaluation_interval=0, alarm_name_zh_tw=分區創建失敗} } 2023-11-27 13:37:03.674 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,13d102653cd2] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000038, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000031, task_operation=execute, latest_execution_start_time=2023-11-27T13:37:03.538+08:00, sub_task_instance_id=10000038}, listMap={}}, executor=10.120.0.1 2023-11-27 13:37:03.677 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=profile_credential_sharing, partition=p1701187200 2023-11-27 13:37:03.679 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:03.680 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:03.681 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `profile_credential_sharing` add partition(partition p1701187200 values less than (1701187200)) 2023-11-27 13:37:03.681 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=6 2023-11-27 13:37:03.682 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=9 2023-11-27 13:37:03.685 INFO 10 --- [pool-281-thread-1,554481c3fd1b426b,74bc4170fc32] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:03.686 INFO 10 --- [pool-281-thread-1,554481c3fd1b426b,74bc4170fc32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:03.688 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-11-27 13:37:03.688 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-11-27 13:37:03.694 INFO 10 --- [pool-281-thread-2,554481c3fd1b426b,593ca3e2f4b2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:03.698 INFO 10 --- [pool-281-thread-1,554481c3fd1b426b,74bc4170fc32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:03.698 INFO 10 --- [pool-281-thread-2,554481c3fd1b426b,593ca3e2f4b2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:03.698 INFO 10 --- [pool-281-thread-2,554481c3fd1b426b,593ca3e2f4b2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:03.719 INFO 10 --- [pool-281-thread-1,554481c3fd1b426b,74bc4170fc32] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:03.719 INFO 10 --- [pool-281-thread-1,554481c3fd1b426b,74bc4170fc32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:03.723 INFO 10 --- [pool-281-thread-1,554481c3fd1b426b,74bc4170fc32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-11-27 13:37:03.724 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2023-11-27 13:37:03.725 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:37:03.731 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,21f56e121b00] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000037, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:37:03.732 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-11-27 13:37:03.738 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:37:03.751 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:37:03.786 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:37:03.802 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:03.803 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000034, status:SUCCESS 2023-11-27 13:37:03.807 INFO 10 --- [pool-281-thread-1,554481c3fd1b426b,74bc4170fc32] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1701021601748952808, broadcast scn:1701021601748952808, saved frozen scn:1701021601748952808 2023-11-27 13:37:03.808 INFO 10 --- [pool-281-thread-1,554481c3fd1b426b,74bc4170fc32] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:darenOB, tenant:sys 2023-11-27 13:37:03.809 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000034, status:SUCCESS 2023-11-27 13:37:03.822 INFO 10 --- [pool-281-thread-2,554481c3fd1b426b,593ca3e2f4b2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:03.833 INFO 10 --- [pool-281-thread-2,554481c3fd1b426b,593ca3e2f4b2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:03.833 INFO 10 --- [pool-281-thread-2,554481c3fd1b426b,593ca3e2f4b2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:03.834 INFO 10 --- [pool-281-thread-2,554481c3fd1b426b,593ca3e2f4b2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-11-27 13:37:03.840 INFO 10 --- [pool-281-thread-2,554481c3fd1b426b,593ca3e2f4b2] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1701021600396785720, broadcast scn:1701021600396785720, saved frozen scn:1701021600396785720 2023-11-27 13:37:03.840 INFO 10 --- [pool-281-thread-2,554481c3fd1b426b,593ca3e2f4b2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:darenOB, tenant:daren 2023-11-27 13:37:03.847 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,0e9ae7c8e53b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000039, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000032, task_operation=execute, latest_execution_start_time=2023-11-27T13:37:03.617+08:00, sub_task_instance_id=10000039}, listMap={}}, executor=10.120.0.1 2023-11-27 13:37:03.849 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,d41e5fe7ff9e] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-11-27 13:37:03.852 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,13d102653cd2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-11-27 13:37:03.854 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,13d102653cd2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster darenOB, trans stat result is empty. 2023-11-27 13:37:03.861 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,d41e5fe7ff9e] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000040, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:37:03.865 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,13d102653cd2] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000038, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:37:03.909 INFO 10 --- [pool-ocp-schedules-20,,3ae02dbf057f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:37:04.004 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,0e9ae7c8e53b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obcluster, sql count is 0. 2023-11-27 13:37:04.004 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,0e9ae7c8e53b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, result is empty. 2023-11-27 13:37:04.008 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,0e9ae7c8e53b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster darenOB, sql count is 0. 2023-11-27 13:37:04.008 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,0e9ae7c8e53b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster darenOB, result is empty. 2023-11-27 13:37:04.014 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.11:62889/api/v1/log/alarms, params:null 2023-11-27 13:37:04.025 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-11-27 13:37:04.031 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,0e9ae7c8e53b] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000039, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:37:04.040 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.88:62889/api/v1/log/alarms, params:null 2023-11-27 13:37:04.047 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2023-11-27 13:37:04.053 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.92:62889/api/v1/log/alarms, params:null 2023-11-27 13:37:04.057 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.12:62889/api/v1/log/alarms, params:null 2023-11-27 13:37:04.058 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-11-27 13:37:04.061 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:37:04.079 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2023-11-27 13:37:04.088 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:37:04.112 INFO 10 --- [pool-ocp-schedules-15,,253bda5db359] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=584494706496503808 2023-11-27 13:37:04.132 INFO 10 --- [pool-ocp-schedules-15,,253bda5db359] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, newSeriesIdsCount=208, maxSyncedSeriesId=585625099342655488 2023-11-27 13:37:04.142 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:04.143 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:04.167 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='profile_credential_sharing' and table_type='BASE TABLE' 2023-11-27 13:37:04.236 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:37:04.252 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.13:62889/api/v1/log/alarms, params:null 2023-11-27 13:37:04.322 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm is revoled, alarmId=19563, status=Inactive, labels=class OcpLabels { {app=OCP, alarm_name=分區創建失敗, app_type=OCP, alarm_name_en_us=partition create failed, alarm_name_zh_cn=分区创建失败, alarm_level=3, alarm_template_id=0, table_name=profile_credential_sharing, alarm_summary_zh_tw=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 分區創建失敗, result=success, alarm_summary_en_us=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing partition create failed, alarm_description_en_us=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing partition create failed partition (p1701187200) create failed, result success., alarm_description_zh_cn=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 分区创建失败 分区（p1701187200）创建失败，结果：success。, alarm_type=partition_create_failed, datasource=Metadb, alarm_status=Inactive, alarm_summary_zh_cn=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 分区创建失败, alarm_scope=Service, alarm_target=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing, ocp_ip=10.120.0.1, alarm_description_zh_tw=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 分區創建失敗 分區（p1701187200）創建失敗，結果：success。, partition_name=p1701187200, alarm_evaluation_interval=0, alarm_name_zh_tw=分區創建失敗} } 2023-11-27 13:37:04.339 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=profile_credential_sharing, partition=p1701273600 2023-11-27 13:37:04.341 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:04.341 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:04.347 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `profile_credential_sharing` add partition(partition p1701273600 values less than (1701273600)) 2023-11-27 13:37:04.360 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.3:62889/api/v1/log/alarms, params:null 2023-11-27 13:37:04.373 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.97:62889/api/v1/log/alarms, params:null 2023-11-27 13:37:04.630 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:04.630 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:04.631 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='profile_credential_sharing' and table_type='BASE TABLE' 2023-11-27 13:37:04.748 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:37:04.778 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:37:04.781 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm is revoled, alarmId=19563, status=Inactive, labels=class OcpLabels { {app=OCP, alarm_name=分區創建失敗, app_type=OCP, alarm_name_en_us=partition create failed, alarm_name_zh_cn=分区创建失败, alarm_level=3, alarm_template_id=0, table_name=profile_credential_sharing, alarm_summary_zh_tw=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 分區創建失敗, result=success, alarm_summary_en_us=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing partition create failed, alarm_description_en_us=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing partition create failed partition (p1701273600) create failed, result success., alarm_description_zh_cn=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 分区创建失败 分区（p1701273600）创建失败，结果：success。, alarm_type=partition_create_failed, datasource=Metadb, alarm_status=Inactive, alarm_summary_zh_cn=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 分区创建失败, alarm_scope=Service, alarm_target=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing, ocp_ip=10.120.0.1, alarm_description_zh_tw=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 分區創建失敗 分區（p1701273600）創建失敗，結果：success。, partition_name=p1701273600, alarm_evaluation_interval=0, alarm_name_zh_tw=分區創建失敗} } 2023-11-27 13:37:04.798 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Metadb, table=profile_credential_sharing, partition=p1700236800 2023-11-27 13:37:04.804 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:04.804 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:04.805 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `profile_credential_sharing` drop partition (p1700236800) 2023-11-27 13:37:04.831 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:37:05.027 INFO 10 --- [alarm-detect-1,,5696d692d075] c.o.o.s.a.core.detect.MetricDetector : alarmType host_unavailable detect time range 1701063354 ~ 1701063414 matched, alarm-level: 1, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={host_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=host_available, match=true, value=0.0, values={1701063360=0.0}, valueCount=1, labels={app=HOST, host=133.197.206.92, host_name=localhost.localdomain, ob_cluster_id=, ob_cluster_name=, svr_ip=133.197.206.92})]}, matched=true, message=null, level=1) 2023-11-27 13:37:05.028 INFO 10 --- [alarm-detect-1,,5696d692d075] c.o.o.s.a.core.detect.MetricDetector : alarmType host_unavailable detect time range 1701063354 ~ 1701063414 matched, alarm-level: 1, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={host_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=host_available, match=true, value=0.0, values={1701063360=0.0}, valueCount=1, labels={app=HOST, host=133.197.206.97, host_name=localhost.localdomain, ob_cluster_id=1, ob_cluster_name=obcluster, svr_ip=133.197.206.97})]}, matched=true, message=null, level=1) 2023-11-27 13:37:05.028 INFO 10 --- [alarm-detect-1,,5696d692d075] c.o.o.s.a.core.detect.MetricDetector : alarmType host_unavailable detect time range 1701063354 ~ 1701063414 matched, alarm-level: 1, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={host_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=host_available, match=true, value=0.0, values={1701063360=0.0}, valueCount=1, labels={app=HOST, host=133.197.206.88, host_name=localhost.localdomain, ob_cluster_id=, ob_cluster_name=, svr_ip=133.197.206.88})]}, matched=true, message=null, level=1) 2023-11-27 13:37:05.028 INFO 10 --- [alarm-detect-1,,5696d692d075] c.o.o.s.a.core.detect.MetricDetector : alarmType host_unavailable detect time range 1701063354 ~ 1701063414 matched, alarm-level: 1, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={host_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=host_available, match=true, value=0.0, values={1701063360=0.0}, valueCount=1, labels={app=HOST, host=133.197.204.3, host_name=localhost.localdomain, ob_cluster_id=10, ob_cluster_name=darenOB, svr_ip=133.197.204.3})]}, matched=true, message=null, level=1) 2023-11-27 13:37:05.029 INFO 10 --- [alarm-detect-1,,5696d692d075] c.o.o.s.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=host_unavailable, target=alarm_template_id=0:host=133.197.206.92-1, currentState=Pending 2023-11-27 13:37:05.034 INFO 10 --- [alarm-detect-1,,5696d692d075] c.o.o.s.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=host_unavailable, target=alarm_template_id=0:host=133.197.206.97-1, currentState=Pending 2023-11-27 13:37:05.034 INFO 10 --- [alarm-detect-1,,5696d692d075] c.o.o.s.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=host_unavailable, target=alarm_template_id=0:host=133.197.206.88-1, currentState=Pending 2023-11-27 13:37:05.034 INFO 10 --- [alarm-detect-1,,5696d692d075] c.o.o.s.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=host_unavailable, target=alarm_template_id=0:host=133.197.204.3-1, currentState=Pending 2023-11-27 13:37:05.108 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063425, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:05.108 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063425, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:05.110 INFO 10 --- [alarm-detect-6,,a514ee35f443] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063354 ~ 1701063414 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063360=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/obproxy, exporter_addr=http://133.197.204.3:62889/metrics/obproxy, exporter_type=OB_PROXY, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:37:05.114 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063425, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:05.114 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063425, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:05.114 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063425, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:05.115 INFO 10 --- [alarm-detect-6,,a514ee35f443] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063354 ~ 1701063414 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063360=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/host, exporter_addr=http://133.197.204.3:62889/metrics/node/host, exporter_type=HOST_MONITOR, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:37:05.115 INFO 10 --- [alarm-detect-6,,a514ee35f443] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063354 ~ 1701063414 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063360=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://133.197.204.3:62889/metrics/stat, exporter_type=HOST_MONITOR, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:37:05.117 INFO 10 --- [alarm-detect-6,,a514ee35f443] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063354 ~ 1701063414 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063360=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/ob/basic, exporter_addr=http://133.197.204.3:62889/metrics/ob/basic, exporter_type=OB_CLUSTER, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:37:05.117 INFO 10 --- [alarm-detect-6,,a514ee35f443] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063354 ~ 1701063414 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063360=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/ob, exporter_addr=http://133.197.204.3:62889/metrics/node/ob, exporter_type=OB_CLUSTER, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:37:05.118 INFO 10 --- [alarm-detect-6,,a514ee35f443] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063354 ~ 1701063414 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063360=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/ob/extra, exporter_addr=http://133.197.204.3:62889/metrics/ob/extra, exporter_type=OB_CLUSTER, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:37:05.118 INFO 10 --- [alarm-detect-6,,a514ee35f443] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063354 ~ 1701063414 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063360=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://133.197.204.3:62888/metrics/stat, exporter_type=HOST_MONITOR, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:37:05.118 INFO 10 --- [alarm-detect-6,,a514ee35f443] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063354 ~ 1701063414 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063360=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/obproxy, exporter_addr=http://133.197.204.3:62889/metrics/node/obproxy, exporter_type=OB_PROXY, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:37:05.118 INFO 10 --- [alarm-detect-6,,a514ee35f443] c.o.o.s.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=monitor_exporter_unavaliable, target=alarm_template_id=0:host=133.197.204.3:exporter=/metrics/obproxy-3, currentState=Pending 2023-11-27 13:37:05.130 INFO 10 --- [alarm-detect-6,,a514ee35f443] c.o.o.s.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=monitor_exporter_unavaliable, target=alarm_template_id=0:host=133.197.204.3:exporter=/metrics/node/host-3, currentState=Pending 2023-11-27 13:37:05.130 INFO 10 --- [alarm-detect-6,,a514ee35f443] c.o.o.s.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=monitor_exporter_unavaliable, target=alarm_template_id=0:host=133.197.204.3:exporter=/metrics/stat-3, currentState=Pending 2023-11-27 13:37:05.131 INFO 10 --- [alarm-detect-6,,a514ee35f443] c.o.o.s.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=monitor_exporter_unavaliable, target=alarm_template_id=0:host=133.197.204.3:exporter=/metrics/ob/basic-3, currentState=Pending 2023-11-27 13:37:05.131 INFO 10 --- [alarm-detect-6,,a514ee35f443] c.o.o.s.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=monitor_exporter_unavaliable, target=alarm_template_id=0:host=133.197.204.3:exporter=/metrics/node/ob-3, currentState=Pending 2023-11-27 13:37:05.131 INFO 10 --- [alarm-detect-6,,a514ee35f443] c.o.o.s.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=monitor_exporter_unavaliable, target=alarm_template_id=0:host=133.197.204.3:exporter=/metrics/ob/extra-3, currentState=Pending 2023-11-27 13:37:05.133 INFO 10 --- [alarm-detect-6,,a514ee35f443] c.o.o.s.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=monitor_exporter_unavaliable, target=alarm_template_id=0:host=133.197.204.3:exporter=/metrics/node/obproxy-3, currentState=Pending 2023-11-27 13:37:05.175 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000030, status:SUCCESS 2023-11-27 13:37:05.176 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000030, status:SUCCESS 2023-11-27 13:37:05.209 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:36:07.0, nextRun=2023-11-27 13:37:05.0, currentDate=Mon Nov 27 13:37:05 CST 2023 2023-11-27 13:37:05.209 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-11-27 13:36:07.0, nextRunTime=2023-11-27 13:37:05.0} 2023-11-27 13:37:05.221 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000031, status:SUCCESS 2023-11-27 13:37:05.223 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000031, status:SUCCESS 2023-11-27 13:37:05.252 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:05.252 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:05.253 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='profile_credential_sharing' and table_type='BASE TABLE' 2023-11-27 13:37:05.255 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000032, status:SUCCESS 2023-11-27 13:37:05.258 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000032, status:SUCCESS 2023-11-27 13:37:05.278 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000033, status:SUCCESS 2023-11-27 13:37:05.280 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000033, status:SUCCESS 2023-11-27 13:37:05.316 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Metadb, table=profile_credential_sharing, partition=p1700323200 2023-11-27 13:37:05.319 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-11-27 13:37:05.319 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:05.319 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:05.319 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:37:05.319 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-11-27 13:37:05.320 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `profile_credential_sharing` drop partition (p1700323200) 2023-11-27 13:37:05.462 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:05.462 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:05.464 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='profile_credential_sharing' and table_type='BASE TABLE' 2023-11-27 13:37:05.488 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Metadb, table=profile_credential_sharing, partition=p1700409600 2023-11-27 13:37:05.488 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:05.488 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:05.489 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `profile_credential_sharing` drop partition (p1700409600) 2023-11-27 13:37:05.552 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:37:05.578 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:05.578 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:05.579 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='profile_credential_sharing' and table_type='BASE TABLE' 2023-11-27 13:37:05.614 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:05.614 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:05.618 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:37:05.619 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:05.619 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:05.620 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2023-11-27 13:37:05.668 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=ocp_cache_info, partition=p1701100800 2023-11-27 13:37:05.669 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:05.669 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:05.671 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_cache_info` add partition(partition p1701100800 values less than (1701100800)) 2023-11-27 13:37:05.813 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000042, operation:EXECUTE, state: RUNNING 2023-11-27 13:37:05.817 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000042, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-11-27T13:37:05.804+08:00, endTime=null} 2023-11-27 13:37:05.833 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:05.833 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:05.835 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='ocp_cache_info' and table_type='BASE TABLE' 2023-11-27 13:37:05.872 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,bdd434128c37] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000042, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000035, task_operation=execute, latest_execution_start_time=2023-11-27T13:37:05.804+08:00, sub_task_instance_id=10000042}, listMap={}}, executor=10.120.0.1 2023-11-27 13:37:05.954 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=ocp_cache_info 2023-11-27 13:37:05.961 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=ocp_cache_info, partition=p1701187200 2023-11-27 13:37:05.970 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:05.971 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:05.976 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_cache_info` add partition(partition p1701187200 values less than (1701187200)) 2023-11-27 13:37:06.013 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:06.127 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:37:06.139 INFO 10 --- [pool-Check obproxy6,554481c3fd1b426b,a326284e35e8] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-11-27 13:37:06.142 WARN 10 --- [pool-Check obproxy6,554481c3fd1b426b,a326284e35e8] c.oceanbase.ocp.odpsdk.OdpJdbcOperator : Failed to check odp connection Connection {driverClassName ='com.oceanbase.jdbc.Driver', connectionProperties ='[user=root@proxysys, url=jdbc:oceanbase://133.197.204.3:2883/?connectProxy=true]'}, error is Failed to obtain JDBC Connection; nested exception is java.sql.SQLNonTransientConnectionException: Could not connect to HostAddress{host='133.197.204.3', port=2883}. Connection refused (Connection refused) 2023-11-27 13:37:06.142 WARN 10 --- [pool-Check obproxy6,554481c3fd1b426b,a326284e35e8] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to connect obproxy server, maybe caused by odp process down, wrong password, or ocp address not in private ip list and odp parameter skip_proxy_sys_private_check is not true 2023-11-27 13:37:06.142 ERROR 10 --- [pool-Check obproxy6,554481c3fd1b426b,a326284e35e8] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.UnexpectedException occurred with code error.obproxy.connection.error, and args [133.197.204.3:2883] 2023-11-27 13:37:06.150 INFO 10 --- [pool-Check obproxy8,554481c3fd1b426b,514f43293e87] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-11-27 13:37:06.158 WARN 10 --- [pool-Check obproxy6,554481c3fd1b426b,a326284e35e8] c.oceanbase.ocp.odpsdk.OdpJdbcOperator : Failed to check odp connection Connection {driverClassName ='com.oceanbase.jdbc.Driver', connectionProperties ='[user=root@proxysys, url=jdbc:oceanbase://133.197.204.3:2883/?connectProxy=true]'}, error is Failed to obtain JDBC Connection; nested exception is java.sql.SQLNonTransientConnectionException: Could not connect to HostAddress{host='133.197.204.3', port=2883}. Connection refused (Connection refused) 2023-11-27 13:37:06.158 WARN 10 --- [pool-Check obproxy6,554481c3fd1b426b,a326284e35e8] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to connect obproxy server, maybe caused by odp process down, wrong password, or ocp address not in private ip list and odp parameter skip_proxy_sys_private_check is not true 2023-11-27 13:37:06.158 ERROR 10 --- [pool-Check obproxy6,554481c3fd1b426b,a326284e35e8] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.UnexpectedException occurred with code error.obproxy.connection.error, and args [133.197.204.3:2883] 2023-11-27 13:37:06.182 INFO 10 --- [pool-Check obproxy6,554481c3fd1b426b,a326284e35e8] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 failed, will not update last active time com.oceanbase.ocp.core.exception.UnexpectedException: [OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=OBPROXY_CONNECTION_ERROR, args=133.197.204.3:2883 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_312] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_312] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_312] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_312] at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.util.ExceptionUtils.throwException(ExceptionUtils.java:90) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.util.ExceptionUtils.unExpected(ExceptionUtils.java:77) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.service.operation.obproxy.SystemOdpOperatorFactory.createOdpOperator(SystemOdpOperatorFactory.java:105) ~[ocp-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.service.operation.obproxy.SystemOdpOperatorFactory.tryToCreateOdpOperator(SystemOdpOperatorFactory.java:86) ~[ocp-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.service.operation.obproxy.ObproxyService.lambda$checkObproxyServers$20(ObproxyService.java:517) ~[ocp-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] 2023-11-27 13:37:06.187 INFO 10 --- [pool-Check obproxy8,554481c3fd1b426b,514f43293e87] c.o.o.s.o.obproxy.ObproxyService : check obproxy 10 success, update last active time to now 2023-11-27 13:37:06.192 INFO 10 --- [pool-Check obproxy10,554481c3fd1b426b,980a6762a425] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-11-27 13:37:06.194 INFO 10 --- [pool-Check obproxy9,554481c3fd1b426b,a5a4464b1d3b] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-11-27 13:37:06.207 INFO 10 --- [pool-Check obproxy7,554481c3fd1b426b,8678495b9648] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-11-27 13:37:06.240 INFO 10 --- [pool-Check obproxy7,554481c3fd1b426b,8678495b9648] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-11-27 13:37:06.244 INFO 10 --- [pool-Check obproxy9,554481c3fd1b426b,a5a4464b1d3b] c.o.o.s.o.obproxy.ObproxyService : check obproxy 11 success, update last active time to now 2023-11-27 13:37:06.245 INFO 10 --- [pool-Check obproxy10,554481c3fd1b426b,980a6762a425] c.o.o.s.o.obproxy.ObproxyService : check obproxy 12 success, update last active time to now 2023-11-27 13:37:06.303 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,bdd434128c37] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000042, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:37:06.343 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:06.343 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:06.344 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='ocp_cache_info' and table_type='BASE TABLE' 2023-11-27 13:37:06.459 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000035, status:SUCCESS 2023-11-27 13:37:06.462 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000035, status:SUCCESS 2023-11-27 13:37:06.485 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=ocp_cache_info 2023-11-27 13:37:06.488 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=ocp_cache_info, partition=p1701273600 2023-11-27 13:37:06.490 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:06.490 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:06.490 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_cache_info` add partition(partition p1701273600 values less than (1701273600)) 2023-11-27 13:37:06.718 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:06.718 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:06.733 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='ocp_cache_info' and table_type='BASE TABLE' 2023-11-27 13:37:06.788 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:37:06.789 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:37:06.825 INFO 10 --- [part-initial-worker,,] c.o.o.s.a.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=ocp_cache_info 2023-11-27 13:37:06.827 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Metadb, table=ocp_cache_info, partition=p1700236800 2023-11-27 13:37:06.827 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:06.827 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:06.828 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_cache_info` drop partition (p1700236800) 2023-11-27 13:37:06.914 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:06.914 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:06.926 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='ocp_cache_info' and table_type='BASE TABLE' 2023-11-27 13:37:06.948 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Metadb, table=ocp_cache_info, partition=p1700323200 2023-11-27 13:37:06.955 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:06.955 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:06.959 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_cache_info` drop partition (p1700323200) 2023-11-27 13:37:07.031 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:07.031 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:07.032 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='ocp_cache_info' and table_type='BASE TABLE' 2023-11-27 13:37:07.048 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167016] Dropping partition, scope=Metadb, table=ocp_cache_info, partition=p1700409600 2023-11-27 13:37:07.054 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:07.055 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:07.061 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_cache_info` drop partition (p1700409600) 2023-11-27 13:37:07.136 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:07.136 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:07.137 INFO 10 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='ocp_cache_info' and table_type='BASE TABLE' 2023-11-27 13:37:07.154 INFO 10 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2023-11-27 13:37:07.766 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:09.155 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:09.458 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.120.0.1:8080 2023-11-27 13:37:09.792 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:37:09.798 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:37:10.104 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063430, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:10.107 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063430, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:10.112 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063430, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:10.115 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063430, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:10.117 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063430, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:10.420 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:11.264 INFO 10 --- [pool-ocp-schedules-8,,6e289b89bc8a] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-11-27 13:37:11.265 INFO 10 --- [pool-Check host status17,,4d2058845a45] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=133.197.206.97, originalStatus=OFFLINE 2023-11-27 13:37:11.265 INFO 10 --- [pool-Check host status18,,0a5a6fe8859a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=133.197.204.3, originalStatus=OFFLINE 2023-11-27 13:37:11.265 INFO 10 --- [pool-Check host status19,,2c114fbdb57e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=133.197.206.92, originalStatus=OFFLINE 2023-11-27 13:37:11.266 INFO 10 --- [pool-Check host status20,,1e711da60839] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=133.197.204.10, originalStatus=ONLINE 2023-11-27 13:37:11.268 INFO 10 --- [pool-Check host status21,,b80b07a8c3fa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=133.197.204.11, originalStatus=ONLINE 2023-11-27 13:37:11.268 INFO 10 --- [pool-Check host status22,,4ec1b7b687c4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=6, ip=133.197.204.12, originalStatus=ONLINE 2023-11-27 13:37:11.268 INFO 10 --- [pool-Check host status23,,726cc5ed4564] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=133.197.204.13, originalStatus=ONLINE 2023-11-27 13:37:11.270 INFO 10 --- [pool-Check host status24,,e8a910cffc63] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=9, ip=133.197.206.88, originalStatus=OFFLINE 2023-11-27 13:37:11.306 INFO 10 --- [pool-Check host status23,,726cc5ed4564] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2023-11-27 13:37:11.328 INFO 10 --- [pool-Check host status17,,4d2058845a45] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:37:11.328 INFO 10 --- [pool-Check host status19,,2c114fbdb57e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:37:11.329 INFO 10 --- [pool-Check host status20,,1e711da60839] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-11-27 13:37:11.353 INFO 10 --- [pool-Check host status24,,e8a910cffc63] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:37:11.375 INFO 10 --- [pool-Check host status18,,0a5a6fe8859a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:37:11.378 INFO 10 --- [pool-Check host status22,,4ec1b7b687c4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2023-11-27 13:37:11.401 INFO 10 --- [pool-Check host status21,,b80b07a8c3fa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-11-27 13:37:11.506 INFO 10 --- [pool-Check host status24,,e8a910cffc63] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.88:62888/api/v1/time, params:null 2023-11-27 13:37:11.510 INFO 10 --- [pool-Check host status22,,4ec1b7b687c4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.12:62888/api/v1/time, params:null 2023-11-27 13:37:11.511 INFO 10 --- [pool-Check host status20,,1e711da60839] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.10:62888/api/v1/time, params:null 2023-11-27 13:37:11.512 INFO 10 --- [pool-Check host status23,,726cc5ed4564] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.13:62888/api/v1/time, params:null 2023-11-27 13:37:11.515 WARN 10 --- [pool-Check host status24,,e8a910cffc63] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check failed. host=9, ip=133.197.206.88, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.88:62888 [/133.197.206.88] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.getHostTime(AgentExecutor.java:127) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.doCheckHostStatus(HostCheckManagerImpl.java:206) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostStatus(HostCheckManagerImpl.java:168) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.88:62888 [/133.197.206.88] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 24 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 33 common frames omitted 2023-11-27 13:37:11.516 INFO 10 --- [pool-Check host status20,,1e711da60839] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=133.197.204.10, timestamp=2023-11-27T00:37:11.509024313-05:00 2023-11-27 13:37:11.516 INFO 10 --- [pool-Check host status22,,4ec1b7b687c4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=6, ip=133.197.204.12, timestamp=2023-11-27T00:37:11.510825673-05:00 2023-11-27 13:37:11.520 INFO 10 --- [pool-Check host status20,,1e711da60839] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-11-27T13:37:11.516+08:00 2023-11-27 13:37:11.522 INFO 10 --- [pool-Check host status23,,726cc5ed4564] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=7, ip=133.197.204.13, timestamp=2023-11-27T00:37:11.506593263-05:00 2023-11-27 13:37:11.524 INFO 10 --- [pool-Check host status19,,2c114fbdb57e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.92:62888/api/v1/time, params:null 2023-11-27 13:37:11.527 WARN 10 --- [pool-Check host status19,,2c114fbdb57e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check failed. host=3, ip=133.197.206.92, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.92:62888 [/133.197.206.92] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.getHostTime(AgentExecutor.java:127) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.doCheckHostStatus(HostCheckManagerImpl.java:206) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostStatus(HostCheckManagerImpl.java:168) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.92:62888 [/133.197.206.92] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 24 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 33 common frames omitted 2023-11-27 13:37:11.527 INFO 10 --- [pool-Check host status22,,4ec1b7b687c4] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=6, lastAvailableTime=2023-11-27T13:37:11.517+08:00 2023-11-27 13:37:11.529 INFO 10 --- [pool-Check host status21,,b80b07a8c3fa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.11:62888/api/v1/time, params:null 2023-11-27 13:37:11.534 INFO 10 --- [pool-Check host status23,,726cc5ed4564] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2023-11-27T13:37:11.522+08:00 2023-11-27 13:37:11.540 INFO 10 --- [pool-Check host status21,,b80b07a8c3fa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=133.197.204.11, timestamp=2023-11-27T00:37:11.534121671-05:00 2023-11-27 13:37:11.540 INFO 10 --- [pool-Check host status20,,1e711da60839] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 11 last available time 2023-11-27T13:37:11.516+08:00 and status AVAILABLE 2023-11-27 13:37:11.544 INFO 10 --- [pool-Check host status21,,b80b07a8c3fa] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-11-27T13:37:11.540+08:00 2023-11-27 13:37:11.550 INFO 10 --- [pool-Check host status22,,4ec1b7b687c4] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 13 last available time 2023-11-27T13:37:11.517+08:00 and status AVAILABLE 2023-11-27 13:37:11.550 INFO 10 --- [pool-Check host status24,,e8a910cffc63] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [9:133.197.206.88], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:37:11.550 INFO 10 --- [pool-Check host status24,,e8a910cffc63] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:37:11.555 INFO 10 --- [pool-Check host status19,,2c114fbdb57e] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [3:133.197.206.92], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:37:11.555 INFO 10 --- [pool-Check host status19,,2c114fbdb57e] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:37:11.556 INFO 10 --- [pool-Check host status23,,726cc5ed4564] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 14 last available time 2023-11-27T13:37:11.522+08:00 and status AVAILABLE 2023-11-27 13:37:11.558 INFO 10 --- [pool-Check host status21,,b80b07a8c3fa] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 12 last available time 2023-11-27T13:37:11.540+08:00 and status AVAILABLE 2023-11-27 13:37:11.561 INFO 10 --- [pool-Check host status19,,2c114fbdb57e] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=3 2023-11-27 13:37:11.561 INFO 10 --- [pool-Check host status19,,2c114fbdb57e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2023-11-27 13:37:11.562 INFO 10 --- [pool-Check host status24,,e8a910cffc63] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=9 2023-11-27 13:37:11.563 INFO 10 --- [pool-Check host status24,,e8a910cffc63] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=9 2023-11-27 13:37:11.567 INFO 10 --- [pool-Check host status17,,4d2058845a45] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.97:62888/api/v1/time, params:null 2023-11-27 13:37:11.570 WARN 10 --- [pool-Check host status17,,4d2058845a45] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check failed. host=1, ip=133.197.206.97, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.97:62888 [/133.197.206.97] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.getHostTime(AgentExecutor.java:127) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.doCheckHostStatus(HostCheckManagerImpl.java:206) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostStatus(HostCheckManagerImpl.java:168) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.97:62888 [/133.197.206.97] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 24 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 33 common frames omitted 2023-11-27 13:37:11.572 INFO 10 --- [pool-Check host status19,,2c114fbdb57e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:37:11.574 INFO 10 --- [pool-Check host status24,,e8a910cffc63] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:37:11.594 INFO 10 --- [pool-Check host status17,,4d2058845a45] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [1:133.197.206.97], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:37:11.594 INFO 10 --- [pool-Check host status17,,4d2058845a45] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:37:11.600 INFO 10 --- [pool-Check host status21,,b80b07a8c3fa] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-11-27T13:37:11.600+08:00 2023-11-27 13:37:11.605 INFO 10 --- [pool-Check host status21,,b80b07a8c3fa] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 12 last available time 2023-11-27T13:37:11.600+08:00 and status AVAILABLE 2023-11-27 13:37:11.616 INFO 10 --- [pool-Check host status22,,4ec1b7b687c4] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=6, lastAvailableTime=2023-11-27T13:37:11.599+08:00 2023-11-27 13:37:11.625 INFO 10 --- [pool-Check host status21,,b80b07a8c3fa] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-11-27 13:37:11.625 INFO 10 --- [pool-Check host status17,,4d2058845a45] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=1 2023-11-27 13:37:11.626 INFO 10 --- [pool-Check host status17,,4d2058845a45] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2023-11-27 13:37:11.643 INFO 10 --- [pool-Check host status18,,0a5a6fe8859a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.3:62888/api/v1/time, params:null 2023-11-27 13:37:11.643 INFO 10 --- [pool-Check host status17,,4d2058845a45] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:37:11.645 WARN 10 --- [pool-Check host status18,,0a5a6fe8859a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check failed. host=2, ip=133.197.204.3, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.204.3:62888 [/133.197.204.3] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.getHostTime(AgentExecutor.java:127) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.doCheckHostStatus(HostCheckManagerImpl.java:206) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostStatus(HostCheckManagerImpl.java:168) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.204.3:62888 [/133.197.204.3] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 24 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 33 common frames omitted 2023-11-27 13:37:11.654 INFO 10 --- [pool-Check host status22,,4ec1b7b687c4] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 13 last available time 2023-11-27T13:37:11.599+08:00 and status AVAILABLE 2023-11-27 13:37:11.660 INFO 10 --- [pool-Check host status20,,1e711da60839] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-11-27T13:37:11.655+08:00 2023-11-27 13:37:11.662 INFO 10 --- [pool-Check host status23,,726cc5ed4564] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2023-11-27T13:37:11.659+08:00 2023-11-27 13:37:11.662 INFO 10 --- [pool-Check host status18,,0a5a6fe8859a] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [2:133.197.204.3], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:37:11.662 INFO 10 --- [pool-Check host status18,,0a5a6fe8859a] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:37:11.670 INFO 10 --- [pool-Check host status20,,1e711da60839] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 11 last available time 2023-11-27T13:37:11.655+08:00 and status AVAILABLE 2023-11-27 13:37:11.693 INFO 10 --- [pool-Check host status18,,0a5a6fe8859a] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=2 2023-11-27 13:37:11.693 INFO 10 --- [pool-Check host status18,,0a5a6fe8859a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2023-11-27 13:37:11.693 INFO 10 --- [pool-Check host status23,,726cc5ed4564] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 14 last available time 2023-11-27T13:37:11.659+08:00 and status AVAILABLE 2023-11-27 13:37:11.694 INFO 10 --- [pool-Check host status22,,4ec1b7b687c4] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=6 2023-11-27 13:37:11.695 INFO 10 --- [pool-Check host status21,,b80b07a8c3fa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-11-27 13:37:11.716 INFO 10 --- [pool-Check host status18,,0a5a6fe8859a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:37:11.718 INFO 10 --- [pool-Check host status20,,1e711da60839] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-11-27 13:37:11.761 INFO 10 --- [pool-Check host status23,,726cc5ed4564] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=7 2023-11-27 13:37:11.806 INFO 10 --- [pool-Check host status22,,4ec1b7b687c4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2023-11-27 13:37:11.821 INFO 10 --- [pool-overview-container-8,,b9bca0bf1208] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:37:11.830 INFO 10 --- [pool-Check host status23,,726cc5ed4564] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2023-11-27 13:37:11.831 INFO 10 --- [pool-Check host status20,,1e711da60839] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-11-27 13:37:11.882 INFO 10 --- [pool-Check host status21,,b80b07a8c3fa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-11-27 13:37:11.893 INFO 10 --- [pool-Check host status20,,1e711da60839] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-11-27 13:37:11.907 INFO 10 --- [pool-Check host status22,,4ec1b7b687c4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-11-27 13:37:11.914 INFO 10 --- [pool-Check host status23,,726cc5ed4564] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-11-27 13:37:11.924 INFO 10 --- [pool-Check host status21,,b80b07a8c3fa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=34 2023-11-27 13:37:11.936 INFO 10 --- [pool-Check host status20,,1e711da60839] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=31 2023-11-27 13:37:11.948 INFO 10 --- [pool-Check host status21,,b80b07a8c3fa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=34 2023-11-27 13:37:11.952 INFO 10 --- [pool-Check host status22,,4ec1b7b687c4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=37 2023-11-27 13:37:11.970 INFO 10 --- [pool-Check host status23,,726cc5ed4564] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2023-11-27 13:37:11.994 INFO 10 --- [pool-Check host status22,,4ec1b7b687c4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=37 2023-11-27 13:37:12.001 INFO 10 --- [pool-Check host status20,,1e711da60839] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=31 2023-11-27 13:37:12.022 INFO 10 --- [pool-Check host status23,,726cc5ed4564] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2023-11-27 13:37:12.033 INFO 10 --- [pool-scanner--1756-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=120 2023-11-27 13:37:12.051 INFO 10 --- [pool-Check host status21,,b80b07a8c3fa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-11-27 13:37:12.073 INFO 10 --- [pool-Check host status23,,726cc5ed4564] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-11-27 13:37:12.085 INFO 10 --- [pool-Check host status22,,4ec1b7b687c4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-11-27 13:37:12.091 INFO 10 --- [pool-Check host status21,,b80b07a8c3fa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=35 2023-11-27 13:37:12.100 INFO 10 --- [pool-Check host status20,,1e711da60839] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-11-27 13:37:12.115 INFO 10 --- [pool-Check host status23,,726cc5ed4564] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2023-11-27 13:37:12.127 INFO 10 --- [pool-Check host status22,,4ec1b7b687c4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=38 2023-11-27 13:37:12.143 INFO 10 --- [pool-Check host status20,,1e711da60839] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=32 2023-11-27 13:37:12.147 INFO 10 --- [pool-Check host status21,,b80b07a8c3fa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=35 2023-11-27 13:37:12.156 INFO 10 --- [pool-Check host status23,,726cc5ed4564] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2023-11-27 13:37:12.166 INFO 10 --- [pool-Check host status22,,4ec1b7b687c4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=38 2023-11-27 13:37:12.225 INFO 10 --- [http-nio-0.0.0.0-8080-exec-7,0e675b3a63694a79,a1107df6fe1d] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.11, traceId=0e675b3a63694a79, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:37:12.226 INFO 10 --- [pool-Check host status23,,726cc5ed4564] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-11-27 13:37:12.230 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:12.271 INFO 10 --- [http-nio-0.0.0.0-8080-exec-7,0e675b3a63694a79,a1107df6fe1d] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.11, traceId=0e675b3a63694a79, duration=46 ms] 2023-11-27 13:37:12.273 INFO 10 --- [pool-Check host status23,,726cc5ed4564] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2023-11-27 13:37:12.294 INFO 10 --- [pool-Check host status20,,1e711da60839] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=32 2023-11-27 13:37:12.296 INFO 10 --- [pool-Check host status22,,4ec1b7b687c4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-11-27 13:37:12.322 INFO 10 --- [pool-Check host status21,,b80b07a8c3fa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-11-27 13:37:12.335 INFO 10 --- [pool-Check host status23,,726cc5ed4564] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2023-11-27 13:37:12.340 INFO 10 --- [pool-Check host status22,,4ec1b7b687c4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=39 2023-11-27 13:37:12.366 INFO 10 --- [pool-Check host status21,,b80b07a8c3fa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=36 2023-11-27 13:37:12.371 INFO 10 --- [pool-Check host status22,,4ec1b7b687c4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=39 2023-11-27 13:37:12.379 INFO 10 --- [pool-Check host status20,,1e711da60839] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-11-27 13:37:12.384 INFO 10 --- [pool-Check host status21,,b80b07a8c3fa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=36 2023-11-27 13:37:12.424 INFO 10 --- [pool-Check host status20,,1e711da60839] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=33 2023-11-27 13:37:12.436 INFO 10 --- [pool-Check host status20,,1e711da60839] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=33 2023-11-27 13:37:12.468 INFO 10 --- [pool-Check host status25,,e81277c622d1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=133.197.206.97, originalStatus=OFFLINE 2023-11-27 13:37:12.471 INFO 10 --- [pool-Check host status26,,e7eab6860ad7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=133.197.204.3, originalStatus=OFFLINE 2023-11-27 13:37:12.477 INFO 10 --- [pool-Check host status25,,e81277c622d1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:37:12.478 INFO 10 --- [pool-Check host status27,,3492fd22d00a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=133.197.206.92, originalStatus=OFFLINE 2023-11-27 13:37:12.479 INFO 10 --- [pool-Check host status28,,0530da359e4d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=133.197.204.10, originalStatus=ONLINE 2023-11-27 13:37:12.479 INFO 10 --- [pool-Check host status26,,e7eab6860ad7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:37:12.480 INFO 10 --- [pool-Check host status29,,052d49abdb7a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=133.197.204.11, originalStatus=ONLINE 2023-11-27 13:37:12.481 INFO 10 --- [pool-Check host status2,,a9385992ed67] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=133.197.204.13, originalStatus=ONLINE 2023-11-27 13:37:12.482 INFO 10 --- [pool-Check host status30,,10bda76d4ad5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=6, ip=133.197.204.12, originalStatus=ONLINE 2023-11-27 13:37:12.482 INFO 10 --- [pool-Check host status3,,b7ca287e5521] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=9, ip=133.197.206.88, originalStatus=OFFLINE 2023-11-27 13:37:12.489 INFO 10 --- [pool-Check host status27,,3492fd22d00a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:37:12.492 INFO 10 --- [pool-Check host status29,,052d49abdb7a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-11-27 13:37:12.493 INFO 10 --- [pool-Check host status2,,a9385992ed67] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2023-11-27 13:37:12.507 INFO 10 --- [pool-Check host status3,,b7ca287e5521] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:37:12.508 INFO 10 --- [pool-Check host status28,,0530da359e4d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-11-27 13:37:12.517 INFO 10 --- [pool-Check host status30,,10bda76d4ad5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2023-11-27 13:37:12.587 INFO 10 --- [pool-Check host status28,,0530da359e4d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.10:62888/api/v1/agent/status, params:null 2023-11-27 13:37:12.603 INFO 10 --- [pool-Check host status28,,0530da359e4d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=133.197.204.10 2023-11-27 13:37:12.603 INFO 10 --- [pool-Check host status28,,0530da359e4d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=133.197.204.10 2023-11-27 13:37:12.603 INFO 10 --- [pool-Check host status28,,0530da359e4d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=133.197.204.10 2023-11-27 13:37:12.623 INFO 10 --- [pool-Check host status26,,e7eab6860ad7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.3:62888/api/v1/agent/status, params:null 2023-11-27 13:37:12.625 WARN 10 --- [pool-Check host status26,,e7eab6860ad7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent status check failed. host=2, ip=133.197.204.3, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.204.3:62888 [/133.197.204.3] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.agentStatus(AgentExecutor.java:622) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostAgentStatus(HostCheckManagerImpl.java:83) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.204.3:62888 [/133.197.204.3] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 23 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 32 common frames omitted 2023-11-27 13:37:12.629 INFO 10 --- [pool-Check host status29,,052d49abdb7a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.11:62888/api/v1/agent/status, params:null 2023-11-27 13:37:12.629 INFO 10 --- [pool-Check host status25,,e81277c622d1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.97:62888/api/v1/agent/status, params:null 2023-11-27 13:37:12.631 INFO 10 --- [pool-Check host status26,,e7eab6860ad7] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [2:133.197.204.3], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:37:12.631 INFO 10 --- [pool-Check host status26,,e7eab6860ad7] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:37:12.633 INFO 10 --- [pool-Check host status26,,e7eab6860ad7] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=2 2023-11-27 13:37:12.633 INFO 10 --- [pool-Check host status26,,e7eab6860ad7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2023-11-27 13:37:12.631 WARN 10 --- [pool-Check host status25,,e81277c622d1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent status check failed. host=1, ip=133.197.206.97, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.97:62888 [/133.197.206.97] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.agentStatus(AgentExecutor.java:622) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostAgentStatus(HostCheckManagerImpl.java:83) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.97:62888 [/133.197.206.97] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 23 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 32 common frames omitted 2023-11-27 13:37:12.642 INFO 10 --- [pool-Check host status26,,e7eab6860ad7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:37:12.642 INFO 10 --- [pool-Check host status29,,052d49abdb7a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=133.197.204.11 2023-11-27 13:37:12.642 INFO 10 --- [pool-Check host status29,,052d49abdb7a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=133.197.204.11 2023-11-27 13:37:12.642 INFO 10 --- [pool-Check host status29,,052d49abdb7a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=133.197.204.11 2023-11-27 13:37:12.669 INFO 10 --- [pool-Check host status25,,e81277c622d1] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [1:133.197.206.97], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:37:12.669 INFO 10 --- [pool-Check host status25,,e81277c622d1] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:37:12.672 INFO 10 --- [pool-Check host status27,,3492fd22d00a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.92:62888/api/v1/agent/status, params:null 2023-11-27 13:37:12.675 INFO 10 --- [pool-Check host status25,,e81277c622d1] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=1 2023-11-27 13:37:12.675 INFO 10 --- [pool-Check host status25,,e81277c622d1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2023-11-27 13:37:12.675 WARN 10 --- [pool-Check host status27,,3492fd22d00a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent status check failed. host=3, ip=133.197.206.92, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.92:62888 [/133.197.206.92] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.agentStatus(AgentExecutor.java:622) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostAgentStatus(HostCheckManagerImpl.java:83) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.92:62888 [/133.197.206.92] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 23 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 32 common frames omitted 2023-11-27 13:37:12.677 INFO 10 --- [pool-Check host status2,,a9385992ed67] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.13:62888/api/v1/agent/status, params:null 2023-11-27 13:37:12.683 INFO 10 --- [pool-Check host status2,,a9385992ed67] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=133.197.204.13 2023-11-27 13:37:12.683 INFO 10 --- [pool-Check host status2,,a9385992ed67] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=133.197.204.13 2023-11-27 13:37:12.683 INFO 10 --- [pool-Check host status2,,a9385992ed67] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=133.197.204.13 2023-11-27 13:37:12.690 INFO 10 --- [pool-Check host status27,,3492fd22d00a] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [3:133.197.206.92], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:37:12.690 INFO 10 --- [pool-Check host status27,,3492fd22d00a] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:37:12.693 INFO 10 --- [pool-Check host status25,,e81277c622d1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:37:12.697 INFO 10 --- [pool-Check host status3,,b7ca287e5521] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.88:62888/api/v1/agent/status, params:null 2023-11-27 13:37:12.699 INFO 10 --- [pool-Check host status27,,3492fd22d00a] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=3 2023-11-27 13:37:12.699 INFO 10 --- [pool-Check host status27,,3492fd22d00a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2023-11-27 13:37:12.700 WARN 10 --- [pool-Check host status3,,b7ca287e5521] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent status check failed. host=9, ip=133.197.206.88, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.88:62888 [/133.197.206.88] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.agentStatus(AgentExecutor.java:622) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostAgentStatus(HostCheckManagerImpl.java:83) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.88:62888 [/133.197.206.88] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 23 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 32 common frames omitted 2023-11-27 13:37:12.702 INFO 10 --- [pool-Check host status30,,10bda76d4ad5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.12:62888/api/v1/agent/status, params:null 2023-11-27 13:37:12.707 INFO 10 --- [pool-Check host status27,,3492fd22d00a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:37:12.708 INFO 10 --- [pool-Check host status3,,b7ca287e5521] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [9:133.197.206.88], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:37:12.708 INFO 10 --- [pool-Check host status3,,b7ca287e5521] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:37:12.714 INFO 10 --- [pool-Check host status30,,10bda76d4ad5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=133.197.204.12 2023-11-27 13:37:12.715 INFO 10 --- [pool-Check host status30,,10bda76d4ad5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=133.197.204.12 2023-11-27 13:37:12.715 INFO 10 --- [pool-Check host status30,,10bda76d4ad5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=133.197.204.12 2023-11-27 13:37:12.717 INFO 10 --- [pool-Check host status3,,b7ca287e5521] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=9 2023-11-27 13:37:12.717 INFO 10 --- [pool-Check host status3,,b7ca287e5521] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=9 2023-11-27 13:37:12.722 INFO 10 --- [pool-Check host status3,,b7ca287e5521] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:37:12.901 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:37:12.911 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:37:12.966 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:37:12.966243+08:00, lastProcessTime=2023-11-27T13:37:03.220547+08:00 2023-11-27 13:37:13.131 INFO 10 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1701063360, nodeCount=164, affectRows=164 2023-11-27 13:37:13.132 INFO 10 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1701063360, nodeCount=164, validCount=164 2023-11-27 13:37:13.170 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:37:12.966243+08:00 2023-11-27 13:37:13.220 INFO 10 --- [pool-metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1701063360, nodeCount=543, affectRows=543 2023-11-27 13:37:13.223 INFO 10 --- [pool-metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1701063360, nodeCount=543, validCount=543 2023-11-27 13:37:13.255 INFO 10 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1701063360, nodeCount=1000, affectRows=1000 2023-11-27 13:37:13.259 INFO 10 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1701063360, nodeCount=1000, validCount=1000 2023-11-27 13:37:13.955 INFO 10 --- [pool-ocp-schedules-6,,1d14e3cbf66a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:37:14.069 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:14.235 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:37:14.838 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:37:14.873 INFO 10 --- [alarm-detect-7,,0459dc95e107] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_cluster_exists_inactive_server detect time range 1701063414 ~ 1701063424 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={inactive_server_count > 0=[ExpressionEvaluateResult(matchValue=0.0, metric=inactive_server_count, match=true, value=1.0, values={1701063420=1.0}, valueCount=1, labels={app=OB, ob_cluster_id=10, ob_cluster_name=darenOB, server_ips=133.197.204.3})]}, matched=true, message=null, level=2) 2023-11-27 13:37:14.874 INFO 10 --- [alarm-detect-7,,0459dc95e107] c.o.o.s.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_cluster_exists_inactive_server, target=alarm_template_id=0:ob_cluster=darenOB-10-2, currentState=Firing 2023-11-27 13:37:14.875 INFO 10 --- [alarm-detect-7,,0459dc95e107] c.o.o.s.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_cluster_exists_inactive_server, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=darenOB-10-2] 2023-11-27 13:37:14.941 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:37:14.948 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:37:15.109 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063435, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:15.111 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063435, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:15.111 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063435, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:15.112 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063435, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:15.118 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063435, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:15.584 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:37:15.641 INFO 10 --- [pool-ocp-schedules-5,,e61183b69a11] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:37:15.641 INFO 10 --- [pool-ocp-schedules-5,,e61183b69a11] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:37:15.855 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:15.911 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:37:15.913 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:37:16.153 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:37:17.192 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:18.683 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:18.914 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:37:18.918 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:37:19.946 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:20.085 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:36:22.0, nextRun=2023-11-27 13:37:20.0, currentDate=Mon Nov 27 13:37:20 CST 2023 2023-11-27 13:37:20.085 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-11-27 13:36:22.0, nextRunTime=2023-11-27 13:37:20.0} 2023-11-27 13:37:20.107 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063440, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:20.107 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063440, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:20.107 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063440, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:20.109 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063440, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:20.110 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063440, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:20.140 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-11-27 13:37:20.140 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:37:20.140 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-11-27 13:37:21.137 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000043, operation:EXECUTE, state: RUNNING 2023-11-27 13:37:21.144 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000043, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-11-27T13:37:21.133+08:00, endTime=null} 2023-11-27 13:37:21.186 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,5bed027540ae] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000043, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000036, task_operation=execute, latest_execution_start_time=2023-11-27T13:37:21.133+08:00, sub_task_instance_id=10000043}, listMap={}}, executor=10.120.0.1 2023-11-27 13:37:21.215 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,5bed027540ae] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-11-27 13:37:21.217 INFO 10 --- [ForkJoinPool-23-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-11-27 13:37:21.265 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,5bed027540ae] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000043, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:37:21.345 INFO 10 --- [ForkJoinPool-23-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:21.345 INFO 10 --- [ForkJoinPool-23-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:21.345 INFO 10 --- [ForkJoinPool-23-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:21.345 INFO 10 --- [ForkJoinPool-23-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:21.347 INFO 10 --- [ForkJoinPool-23-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:21.347 INFO 10 --- [ForkJoinPool-23-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:21.347 INFO 10 --- [ForkJoinPool-23-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT MAX(CASE `name` WHEN 'cluster' THEN `info` ELSE '' END ) `cluster`, MAX(CASE `name` WHEN 'merge_status' THEN `info` ELSE '' END ) `merge_status`, MAX(CASE `name` WHEN 'global_broadcast_version' THEN `value` ELSE 0 END ) `global_broadcast_version`, MAX(CASE `name` WHEN 'frozen_version' THEN `value` ELSE 0 END ) `frozen_version`, MAX(CASE `name` WHEN 'last_merged_version' THEN `value` ELSE 0 END ) `last_merged_version`, MAX(CASE `name` WHEN 'frozen_time' THEN `value` ELSE 0 END ) `frozen_time`, MAX(CASE `name` WHEN 'is_merge_error' THEN `value` ELSE 0 END ) `merge_error` FROM oceanbase.__all_zone WHERE `zone` = '' 2023-11-27 13:37:21.362 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:21.363 INFO 10 --- [ForkJoinPool-23-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:21.363 INFO 10 --- [ForkJoinPool-23-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:21.364 INFO 10 --- [ForkJoinPool-23-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select column_value from oceanbase.__all_core_table where table_name ='__all_global_stat' and column_name = 'frozen_version' 2023-11-27 13:37:21.366 INFO 10 --- [ForkJoinPool-23-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:21.366 INFO 10 --- [ForkJoinPool-23-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:21.366 INFO 10 --- [ForkJoinPool-23-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, MAX(CASE `name` WHEN 'region' THEN `info` ELSE '' END ) `region`, MAX(CASE `name` WHEN 'idc' THEN `info` ELSE '' END ) `idc`, MAX(CASE `name` WHEN 'status' THEN `info` ELSE '' END ) `status`, MAX(CASE `name` WHEN 'merge_status' THEN `info` ELSE '' END ) `merge_status`, MAX(CASE `name` WHEN 'broadcast_version' THEN `value` ELSE 0 END ) `broadcast_version`, MAX(CASE `name` WHEN 'all_merged_version' THEN `value` ELSE 0 END ) `all_merged_version`, MAX(CASE `name` WHEN 'last_merged_version' THEN `value` ELSE 0 END ) `last_merged_version`, MAX(CASE `name` WHEN 'merge_start_time' THEN `value` ELSE 0 END ) `merge_start_time`, MAX(CASE `name` WHEN 'last_merged_time' THEN `value` ELSE 0 END ) `last_merged_time`, MAX(CASE `name` WHEN 'is_merge_timeout' THEN `value` ELSE 0 END ) `merge_timeout` FROM oceanbase.__all_zone WHERE `zone` <> '' GROUP BY `zone` 2023-11-27 13:37:21.368 INFO 10 --- [ForkJoinPool-23-worker-2,,] s.t.b.c.CollectAllClustersCompactionTask : compaction collect success, cluster=ObClusterOverview(id=1, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"133.197.206.97:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1700793672036000}, createTime=2022-11-29T14:52:53+08:00, creator=admin, obVersion=3.1.4, tenantCount=3) 2023-11-27 13:37:21.503 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000036, status:SUCCESS 2023-11-27 13:37:21.504 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000036, status:SUCCESS 2023-11-27 13:37:21.784 INFO 10 --- [pool-overview-container-8,,c8eb302d92d4] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-11-27 13:37:21.784 INFO 10 --- [pool-overview-container-8,,c8eb302d92d4] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:37:21.867 INFO 10 --- [pool-scanner--1849-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=112 2023-11-27 13:37:21.918 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:37:21.921 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:37:22.836 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:37:22.835207+08:00, lastProcessTime=2023-11-27T13:37:12.966243+08:00 2023-11-27 13:37:22.843 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:22.955 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.ChannelAggregator : aggregate, pre groupAggregator is null, reset it, channelId=100000, groupByKey=0:ob_cluster_exists_inactive_server:2, alarmId=44798, alarmType=ob_cluster_exists_inactive_server, alarmTarget=alarm_template_id=0:ob_cluster=darenOB-10 2023-11-27 13:37:22.957 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.ChannelAggregator : aggregate, pre groupAggregator is null, reset it, channelId=100001, groupByKey=0:ob_cluster_exists_inactive_server:2, alarmId=44798, alarmType=ob_cluster_exists_inactive_server, alarmTarget=alarm_template_id=0:ob_cluster=darenOB-10 2023-11-27 13:37:22.957 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:37:22.835207+08:00 2023-11-27 13:37:23.395 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.120.0.1:8080 2023-11-27 13:37:23.612 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.120.0.1:8080 2023-11-27 13:37:23.860 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.120.0.1:8080 2023-11-27 13:37:23.875 INFO 10 --- [pool-ocp-schedules-15,,3438af686817] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-11-27 13:37:23.979 INFO 10 --- [pool-ocp-schedules-2,,0380ede6886e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:37:24.235 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:37:24.290 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:24.480 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.120.0.1:8080 2023-11-27 13:37:24.832 INFO 10 --- [alarm-detect-7,,9790e27ed6a2] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_cluster_exists_inactive_server detect time range 1701063424 ~ 1701063434 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={inactive_server_count > 0=[ExpressionEvaluateResult(matchValue=0.0, metric=inactive_server_count, match=true, value=1.0, values={1701063430=1.0}, valueCount=1, labels={app=OB, ob_cluster_id=10, ob_cluster_name=darenOB, server_ips=133.197.204.3})]}, matched=true, message=null, level=2) 2023-11-27 13:37:24.941 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:37:24.941 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:37:24.949 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:37:24.969 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:37:25.016 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:37:25.107 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063445, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:25.107 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063445, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:25.110 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063445, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:25.113 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063445, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:25.118 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063445, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:25.590 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:37:25.717 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:36:26.0, nextRun=2023-11-27 13:37:25.0, currentDate=Mon Nov 27 13:37:25 CST 2023 2023-11-27 13:37:25.718 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-11-27 13:36:26.0, nextRunTime=2023-11-27 13:37:25.0} 2023-11-27 13:37:25.807 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-11-27 13:37:25.808 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:37:25.808 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-11-27 13:37:25.823 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:26.167 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:37:26.267 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000044, operation:EXECUTE, state: RUNNING 2023-11-27 13:37:26.270 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000044, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-11-27T13:37:26.264+08:00, endTime=null} 2023-11-27 13:37:26.327 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9431ca1f394f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000044, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000037, task_operation=execute, latest_execution_start_time=2023-11-27T13:37:26.258+08:00, sub_task_instance_id=10000044}, listMap={host_ids=[1, 2, 3, 4, 5, 6, 7, 9]}}, executor=10.120.0.1 2023-11-27 13:37:26.451 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9431ca1f394f] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270871, maxInactiveSeconds=180 2023-11-27 13:37:26.451 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9431ca1f394f] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270871, maxInactiveSeconds=180 2023-11-27 13:37:26.451 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9431ca1f394f] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270871, maxInactiveSeconds=180 2023-11-27 13:37:26.451 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9431ca1f394f] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270871, maxInactiveSeconds=180 2023-11-27 13:37:26.451 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9431ca1f394f] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270871, maxInactiveSeconds=180 2023-11-27 13:37:26.451 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9431ca1f394f] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270871, maxInactiveSeconds=180 2023-11-27 13:37:26.451 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9431ca1f394f] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270871, maxInactiveSeconds=180 2023-11-27 13:37:26.451 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9431ca1f394f] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270871, maxInactiveSeconds=180 2023-11-27 13:37:26.451 WARN 10 --- [pool-subtask-executor31,554481c3fd1b426b,9431ca1f394f] c.o.o.s.t.b.host.CheckExporterTask : Host [id=1, ip=133.197.206.97, status=OFFLINE] not online, skip check exporter 2023-11-27 13:37:26.451 WARN 10 --- [pool-subtask-executor31,554481c3fd1b426b,9431ca1f394f] c.o.o.s.t.b.host.CheckExporterTask : Host [id=2, ip=133.197.204.3, status=OFFLINE] not online, skip check exporter 2023-11-27 13:37:26.451 WARN 10 --- [pool-subtask-executor31,554481c3fd1b426b,9431ca1f394f] c.o.o.s.t.b.host.CheckExporterTask : Host [id=3, ip=133.197.206.92, status=OFFLINE] not online, skip check exporter 2023-11-27 13:37:26.454 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9431ca1f394f] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=133.197.204.10] desired exporter count: 8, actual exporter count: 8 2023-11-27 13:37:26.459 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9431ca1f394f] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=133.197.204.11] desired exporter count: 8, actual exporter count: 8 2023-11-27 13:37:26.462 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9431ca1f394f] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=6, ip=133.197.204.12] desired exporter count: 8, actual exporter count: 8 2023-11-27 13:37:26.465 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9431ca1f394f] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=7, ip=133.197.204.13] desired exporter count: 8, actual exporter count: 8 2023-11-27 13:37:26.465 WARN 10 --- [pool-subtask-executor31,554481c3fd1b426b,9431ca1f394f] c.o.o.s.t.b.host.CheckExporterTask : Host [id=9, ip=133.197.206.88, status=OFFLINE] not online, skip check exporter 2023-11-27 13:37:26.477 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9431ca1f394f] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000044, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:37:26.846 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000037, status:SUCCESS 2023-11-27 13:37:26.848 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000037, status:SUCCESS 2023-11-27 13:37:27.021 INFO 10 --- [http-nio-0.0.0.0-8080-exec-9,5baa379ea04d4785,e8335ea5ef47] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.13, traceId=5baa379ea04d4785, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:37:27.074 INFO 10 --- [http-nio-0.0.0.0-8080-exec-9,5baa379ea04d4785,e8335ea5ef47] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.13, traceId=5baa379ea04d4785, duration=59 ms] 2023-11-27 13:37:27.187 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:27.949 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:37:27.950 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:37:28.581 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:29.070 INFO 10 --- [pool-prometheus-exporter-check2,,1c5bd6d98eb3] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/stat, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:37:29.070 WARN 10 --- [pool-prometheus-exporter-check2,,1c5bd6d98eb3] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/stat is inactive 2023-11-27 13:37:29.071 INFO 10 --- [pool-prometheus-exporter-check2,,1c5bd6d98eb3] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62888/metrics/stat, message=java.net.ConnectException: Connection refused: /133.197.204.3:62888 2023-11-27 13:37:29.071 WARN 10 --- [pool-prometheus-exporter-check2,,1c5bd6d98eb3] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62888/metrics/stat is inactive 2023-11-27 13:37:29.073 INFO 10 --- [pool-prometheus-exporter-check2,,1c5bd6d98eb3] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/node/host, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:37:29.073 WARN 10 --- [pool-prometheus-exporter-check2,,1c5bd6d98eb3] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/node/host is inactive 2023-11-27 13:37:29.075 INFO 10 --- [pool-prometheus-exporter-check2,,1c5bd6d98eb3] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/obproxy, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:37:29.075 WARN 10 --- [pool-prometheus-exporter-check2,,1c5bd6d98eb3] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/obproxy is inactive 2023-11-27 13:37:29.080 INFO 10 --- [pool-prometheus-exporter-check2,,1c5bd6d98eb3] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/node/obproxy, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:37:29.080 WARN 10 --- [pool-prometheus-exporter-check2,,1c5bd6d98eb3] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/node/obproxy is inactive 2023-11-27 13:37:29.083 INFO 10 --- [pool-prometheus-exporter-check2,,1c5bd6d98eb3] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/ob/extra, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:37:29.083 WARN 10 --- [pool-prometheus-exporter-check2,,1c5bd6d98eb3] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/ob/extra is inactive 2023-11-27 13:37:29.086 INFO 10 --- [pool-prometheus-exporter-check2,,1c5bd6d98eb3] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/node/ob, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:37:29.086 WARN 10 --- [pool-prometheus-exporter-check2,,1c5bd6d98eb3] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/node/ob is inactive 2023-11-27 13:37:29.089 INFO 10 --- [pool-prometheus-exporter-check2,,1c5bd6d98eb3] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/ob/basic, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:37:29.089 WARN 10 --- [pool-prometheus-exporter-check2,,1c5bd6d98eb3] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/ob/basic is inactive 2023-11-27 13:37:30.105 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063450, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:30.105 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063450, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:30.105 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063450, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:30.108 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063450, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:30.110 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063450, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:30.164 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:36:33.0, nextRun=2023-11-27 13:37:30.0, currentDate=Mon Nov 27 13:37:30 CST 2023 2023-11-27 13:37:30.165 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-11-27 13:36:33.0, nextRunTime=2023-11-27 13:37:30.0} 2023-11-27 13:37:30.297 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-11-27 13:37:30.297 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:37:30.297 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:37:30.297 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-11-27 13:37:30.337 INFO 10 --- [http-nio-0.0.0.0-8080-exec-1,7551472bed4947f8,cf1672434ea4] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.12, traceId=7551472bed4947f8, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:37:30.342 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:37:02.0, nextRun=2023-11-27 13:37:30.0, currentDate=Mon Nov 27 13:37:30 CST 2023 2023-11-27 13:37:30.342 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-11-27 13:37:02.0, nextRunTime=2023-11-27 13:37:30.0} 2023-11-27 13:37:30.409 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-11-27 13:37:30.409 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:37:30.409 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-11-27 13:37:30.423 INFO 10 --- [http-nio-0.0.0.0-8080-exec-1,7551472bed4947f8,cf1672434ea4] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.12, traceId=7551472bed4947f8, duration=87 ms] 2023-11-27 13:37:30.447 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000045, operation:EXECUTE, state: RUNNING 2023-11-27 13:37:30.465 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000045, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-11-27T13:37:30.439+08:00, endTime=null} 2023-11-27 13:37:30.490 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000046, operation:EXECUTE, state: RUNNING 2023-11-27 13:37:30.527 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000046, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-11-27T13:37:30.487+08:00, endTime=null} 2023-11-27 13:37:30.549 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000045, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=8000038, task_operation=execute, latest_execution_start_time=2023-11-27T13:37:30.435+08:00, sub_task_instance_id=10000045}, listMap={cluster_ids=[1, 2]}}, executor=10.120.0.1 2023-11-27 13:37:30.554 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:30.635 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000046, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=8000038, task_operation=execute, latest_execution_start_time=2023-11-27T13:37:30.487+08:00, sub_task_instance_id=10000046}, listMap={cluster_ids=[1, 2]}}, executor=10.120.0.1 2023-11-27 13:37:30.672 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:30.684 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:30.685 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:30.691 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:30.709 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:30.709 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:30.713 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, tenant_name, (CASE WHEN tenant_id = 1 THEN 'SYS' ELSE 'USER' END) AS tenant_type, (CASE WHEN compatibility_mode = 1 THEN 'ORACLE' ELSE 'MYSQL' END) AS compatibility_mode, zone_list, primary_zone, locality, status, locked, read_only, in_recyclebin, TIMESTAMPDIFF(SECOND, gmt_create, now()) AS exist_seconds FROM oceanbase.__all_tenant 2023-11-27 13:37:30.722 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:30.722 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:30.725 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1] 2023-11-27 13:37:30.738 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:30.739 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:30.741 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1] 2023-11-27 13:37:30.755 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:30.755 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:30.756 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1001] 2023-11-27 13:37:30.790 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:30.790 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:30.792 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1001] 2023-11-27 13:37:30.804 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:30.804 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:30.805 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1002] 2023-11-27 13:37:30.827 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:30.827 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:30.827 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:30.833 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1002] 2023-11-27 13:37:30.837 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:30.837 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:30.840 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:30.854 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:30.854 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:30.857 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-11-27 13:37:30.859 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-11-27 13:37:30.869 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:30.869 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:30.870 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-11-27 13:37:30.886 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:30.887 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:30.888 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-11-27 13:37:30.900 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:30.900 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:30.901 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-11-27 13:37:30.914 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:37:30.915 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:30.915 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:30.917 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-11-27 13:37:30.929 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:30.933 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:30.933 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:30.935 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:30.937 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:30.937 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:30.938 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-11-27 13:37:30.940 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-11-27 13:37:30.950 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:37:30.953 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:37:30.959 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:37:30.960 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:30.960 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:30.961 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:30.961 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:30.963 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:30.963 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:30.963 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:37:30.997 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:37:30.999 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:31.000 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:31.000 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:31.001 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:31.002 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:31.002 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:31.003 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1001, ALTER_TENANT_LOCALITY] 2023-11-27 13:37:31.026 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:37:31.027 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:31.028 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:31.028 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:31.028 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:31.031 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:31.031 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:31.032 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1001, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:37:31.047 INFO 10 --- [http-nio-0.0.0.0-8080-exec-2,94840c5565b64a39,e2f64865ef51] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.12, traceId=94840c5565b64a39, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:37:31.074 INFO 10 --- [http-nio-0.0.0.0-8080-exec-2,94840c5565b64a39,e2f64865ef51] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.12, traceId=94840c5565b64a39, duration=28 ms] 2023-11-27 13:37:31.084 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:37:31.086 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:31.093 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:31.093 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:31.100 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:31.116 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:31.116 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:31.119 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-11-27 13:37:31.130 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:37:31.131 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:31.132 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:31.132 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:31.132 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:31.142 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:31.142 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:31.143 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-11-27 13:37:31.177 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:37:31.178 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:31.178 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:31.178 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:31.178 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:31.186 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:31.186 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:31.186 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:37:31.187 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-11-27 13:37:31.189 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:37:31.190 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:31.191 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:31.191 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:31.192 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:31.197 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:31.197 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:31.198 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:37:31.250 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.120.0.1 2023-11-27 13:37:31.282 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:37:31.288 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:31.290 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:31.290 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:31.291 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:31.301 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:31.302 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:31.303 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-11-27 13:37:31.305 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:37:31.306 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:31.308 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:31.308 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:31.309 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:31.343 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:31.343 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:31.347 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1] 2023-11-27 13:37:31.362 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:37:31.370 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:37:31.379 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:31.383 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:31.383 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:31.384 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:31.388 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:31.388 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:31.389 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:37:31.390 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-11-27 13:37:31.421 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.120.0.1 2023-11-27 13:37:31.433 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-11-27 13:37:31.435 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:31.435 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:31.436 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1001] 2023-11-27 13:37:31.450 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:37:31.455 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-11-27 13:37:31.457 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:31.457 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:31.459 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1002] 2023-11-27 13:37:31.474 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:37:31.486 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-11-27 13:37:31.486 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-11-27 13:37:31.486 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=obcluster, obClusterId=1 2023-11-27 13:37:31.515 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b0ea33c25281] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000045, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:37:31.524 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:37:31.525 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:31.526 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:31.526 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:31.529 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:31.559 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:31.559 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:31.562 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-11-27 13:37:31.577 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:37:31.593 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-11-27 13:37:31.594 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:31.594 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:31.599 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-11-27 13:37:31.613 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:37:31.617 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-11-27 13:37:31.617 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-11-27 13:37:31.617 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=darenOB, obClusterId=10 2023-11-27 13:37:31.623 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000047, operation:EXECUTE, state: RUNNING 2023-11-27 13:37:31.637 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000047, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-11-27T13:37:31.612+08:00, endTime=null} 2023-11-27 13:37:31.641 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,e5f70de27a5b] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000046, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:37:31.756 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,26d1c0bd6f4e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000047, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000039, task_operation=execute, latest_execution_start_time=2023-11-27T13:37:31.611+08:00, sub_task_instance_id=10000047}, listMap={}}, executor=10.120.0.1 2023-11-27 13:37:31.756 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,26d1c0bd6f4e] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-11-27 13:37:31.796 INFO 10 --- [pool-overview-container-5,,9af9bf99e19f] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:37:31.810 INFO 10 --- [pool-436-thread-2,554481c3fd1b426b,60f17513f27f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:darenOB, tenant:daren 2023-11-27 13:37:31.814 INFO 10 --- [pool-436-thread-1,554481c3fd1b426b,b1f4ccdf0fdd] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:darenOB, tenant:sys 2023-11-27 13:37:31.930 INFO 10 --- [pool-scanner--1942-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=112 2023-11-27 13:37:31.938 INFO 10 --- [pool-436-thread-1,554481c3fd1b426b,b1f4ccdf0fdd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:31.938 INFO 10 --- [pool-436-thread-1,554481c3fd1b426b,b1f4ccdf0fdd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:31.938 INFO 10 --- [pool-436-thread-1,554481c3fd1b426b,b1f4ccdf0fdd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:31.939 INFO 10 --- [pool-436-thread-1,554481c3fd1b426b,b1f4ccdf0fdd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:31.950 INFO 10 --- [pool-436-thread-1,554481c3fd1b426b,b1f4ccdf0fdd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:31.950 INFO 10 --- [pool-436-thread-1,554481c3fd1b426b,b1f4ccdf0fdd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:31.951 INFO 10 --- [pool-436-thread-1,554481c3fd1b426b,b1f4ccdf0fdd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-11-27 13:37:31.960 INFO 10 --- [pool-436-thread-2,554481c3fd1b426b,60f17513f27f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:31.961 INFO 10 --- [pool-436-thread-2,554481c3fd1b426b,60f17513f27f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:31.961 INFO 10 --- [pool-436-thread-2,554481c3fd1b426b,60f17513f27f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:31.964 INFO 10 --- [pool-436-thread-2,554481c3fd1b426b,60f17513f27f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:31.971 INFO 10 --- [pool-436-thread-2,554481c3fd1b426b,60f17513f27f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:31.971 INFO 10 --- [pool-436-thread-2,554481c3fd1b426b,60f17513f27f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:31.973 INFO 10 --- [pool-436-thread-2,554481c3fd1b426b,60f17513f27f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-11-27 13:37:31.979 INFO 10 --- [pool-436-thread-2,554481c3fd1b426b,60f17513f27f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1701021600396785720, broadcast scn:1701021600396785720, saved frozen scn:1701021600396785720 2023-11-27 13:37:31.979 INFO 10 --- [pool-436-thread-2,554481c3fd1b426b,60f17513f27f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:darenOB, tenant:daren 2023-11-27 13:37:31.982 INFO 10 --- [pool-436-thread-1,554481c3fd1b426b,b1f4ccdf0fdd] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1701021601748952808, broadcast scn:1701021601748952808, saved frozen scn:1701021601748952808 2023-11-27 13:37:31.982 INFO 10 --- [pool-436-thread-1,554481c3fd1b426b,b1f4ccdf0fdd] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:darenOB, tenant:sys 2023-11-27 13:37:31.998 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,26d1c0bd6f4e] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-11-27 13:37:32.010 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,26d1c0bd6f4e] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000047, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:37:32.484 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000038, status:SUCCESS 2023-11-27 13:37:32.486 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000038, status:SUCCESS 2023-11-27 13:37:32.505 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000039, status:SUCCESS 2023-11-27 13:37:32.507 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000039, status:SUCCESS 2023-11-27 13:37:32.765 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:37:32.761126+08:00, lastProcessTime=2023-11-27T13:37:22.835207+08:00 2023-11-27 13:37:33.045 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:37:32.761126+08:00 2023-11-27 13:37:33.095 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:33.877 INFO 10 --- [http-nio-0.0.0.0-8080-exec-3,3a001e20e7a0462d,ee843ea4453f] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.11, traceId=3a001e20e7a0462d, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:37:33.892 INFO 10 --- [http-nio-0.0.0.0-8080-exec-3,3a001e20e7a0462d,ee843ea4453f] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.11, traceId=3a001e20e7a0462d, duration=16 ms] 2023-11-27 13:37:33.953 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:37:33.955 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:37:33.977 INFO 10 --- [pool-prometheus-cache-monitor1,,d1faaa178c4f] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1707,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":1707,"seriesIdKeySize":87133,"secondWriteQueue":0,"minuteWriteCacheSize":32684} 2023-11-27 13:37:33.977 INFO 10 --- [pool-prometheus-cache-monitor1,,d1faaa178c4f] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_zone_current_timestamp=38, ob_server_resource_memory_assigned_bytes=21, tcp_retrans=30, ob_unit_num=21, go_memstats_mspan_sys_bytes=17, go_memstats_stack_sys_bytes=17, exporter_export_metrics_bytes_total=28, node_filesystem_size_bytes=87, http_server_requests_seconds_count=40, ob_compaction_frozen_time=18, ob_system_event_total_waits=34, monagent_input_sql_duration_seconds_sum=60, observer_fd_count=20, ob_plan_cache_hit_total=28, processor_process_metrics_total=22, node_filesystem_readonly=86, ob_server_resource_disk_in_use_bytes=19, go_memstats_frees_total=17, sql_audit_input_collect_data_jumped_total=12, io_await=60, host_available=13, go_memstats_mallocs_total=17, mysql_output_write_sql_failed_count=23, ob_unit_config_max_memory_bytes=45, io_util=60, ob_system_event_detail_total_timeouts=782, ob_tenant_disk_v4_data_size=18, ob_latch_sleep_total=326, ob_tenant_unit_min_memory=41, ob_memstore_total_bytes=28, process_resident_memory_bytes=17, ob_cache_size_bytes=47, ob_tenant_unit_max_memory=41, ob_tenant_assigned_cpu_total=25, input_collect_metrics_total=29, ob_tenant_server_available_disk_free=28, ob_tenant_server_avg_cpu_used_percentage=23, node_disk_io_time_weighted_seconds_total=39, node_disk_write_time_seconds_total=39, cpu_count=30, output_write_metrics_total=15, ob_cluster_sync=175, process_open_fds=17, odp_transaction_total=22, ob_system_event_detail_time_waited_us=782, process_max_fds=17, ob_server_resource_cpu_assigned_percent=20, ocp_alarm_detect_duration_ms_count=82, go_memstats_heap_inuse_bytes=17, ob_tenant_server_max_cpu_used_percentage=23, proxy_memory_used_bytes=30, go_memstats_mcache_sys_bytes=17, ob_tenant_disk_v4_log_size=18, node_memory_memfree_bytes=29, process_exists=38, go_gc_duration_seconds=22, ob_server_resource_disk_bytes=21, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=27, node_memory_cached_bytes=29, ob_table_num=24, ob_disk_free_bytes=21, node_filesystem_avail_bytes=87, ob_backup_storage_capacity_bytes=349, ob_active_session_num=27, ob_unit_config_max_iops=45, ob_plan_cache_memory_bytes=28, node_network_transmit_errs_total=65, go_memstats_last_gc_time_seconds=17, go_gc_duration_seconds_count=17, monitor_exporter_avaliable=78, ob_tenant_server_max_memory_used_percentage=23, node_disk_read_bytes_total=39, mysql_output_write_sql_milliseconds_summary_count=25, ob_compaction_is_suspended=18, ob_tenant_unit_log_disk_size=22, go_memstats_lookups_total=17, mysql_output_write_sql_count=25, log_tailer_count=17, host_agent_process_status=21, ob_server_resource_cpu=21, go_memstats_alloc_bytes=17, ob_server_num=58, node_cpu_seconds_total_native=47, ob_tenant_unit_count=39, ob_session_all_num=25, ob_tenant_resource_min_memory=25, ob_memstore_freeze_times=28, ob_system_event_time_waited=34, ob_tenant_resource_min_cpu=25, ob_cluster_status_check=49, monagent_plugin_execute_seconds_total=41, node_net_bandwidth_bps=45, monagent_oceanbase_db_pool_idle=7, go_memstats_heap_idle_bytes=17, go_memstats_gc_sys_bytes=17, ob_server_resource_memory_bytes=21, ob_tenant_sys_partition_required_size=12, ob_latch_get_total=326, go_info=18, ob_tenant_assigned_mem_assigned=25, log_tailer_processed_file_count=24, node_load15=29, ob_tenant_server_required_size=35, ob_unit_config_min_iops=45, ob_server_partition_replica_limit=7, ob_tenant_unit_max_cpu=41, ob_latch_wait_seconds_total=326, node_memory_buffers_bytes=29, go_memstats_heap_released_bytes=17, node_network_receive_bytes_total=65, ob_memstore_freeze_trigger_bytes=28, ob_waitevent_wait_seconds_total=28, ob_server_stopped_duration_seconds=20, exposer_request_latencies_count=8, ob_role_is_rootserver=20, go_memstats_next_gc_bytes=17, mysql_output_context_timeout_discard_metrics_count=17, ob_session_active_num=25, node_filesystem_files_free=109, ob_memtable_max_snapshot_duration_seconds=27, node_disk_read_time_seconds_total=39, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=22, process_virtual_memory_bytes=17, mysql_output_write_sql_retry_count=23, ob_tenant_assigned_cpu_assigned=25, ob_tenant_unit_min_cpu=41, ob_unit_config_max_cpu=45, ob_unit_config_max_session_num=18, http_request_milliseconds_summary_count=79, monagent_oceanbase_audit_timeinfo_cache_size=7, net_connectivity=10, monagent_input_sql_duration_seconds_bucket=70, ob_latch_immediate_get_total=326, node_cpu_seconds_total=37, http_request_milliseconds_summary=82, ob_latch_immediate_miss_total=326, ob_index_error_num=16, ob_tenant_server_available_disk_total=28, ob_latch_spin_get_total=326, ob_tenant_memory_hold_bytes=18, go_memstats_buck_hash_sys_bytes=17, monagent_pipeline_report_metrics_total=18, mysql_output_write_sql_milliseconds_summary_sum=25, ob_tenant_disk_total_size=25, exposer_transferred_bytes_total=8, odp_sql_request_total=67, mysql_output_write_task_size=24, ob_tenant_context_memory_count=339, node_disk_reads_completed_total=39, exporter_export_metrics_total=28, ob_sysstat=129, ob_partition_num=13, node_ntp_offset_seconds=29, ob_latch_miss_total=326, go_memstats_heap_sys_bytes=17, ocp_meta_db_health=4, ob_disk_total_bytes=21, ob_zone_stat=38, go_memstats_heap_objects=17, go_memstats_mcache_inuse_bytes=17, go_memstats_other_sys_bytes=17, go_threads=17, http_request_milliseconds_summary_sum=79, ob_memstore_active_bytes=28, http_server_requests_seconds_sum=40, go_memstats_alloc_bytes_total=17, ob_tenant_server_partition_replica_count=35, ob_tenant_assigned_mem_total=25, node_memory_memtotal_bytes=29, node_network_transmit_drop_total=65, ob_system_event_detail_total_waits=782, odp_used_connections=28, ob_tenant_task_max_duration_seconds=25, backup_storage_capacity_used_percent=14, exposer_request_latencies=11, ob_waitevent_wait_total=28, odp_sql_cost_total=66, ob_tenant500_memory_used_bytes=20, node_network_transmit_bytes_total=65, ob_compaction_is_error=18, node_network_receive_drop_total=65, ob_server_partition_replica_count=7, monagent_pipeline_execute_total=27, node_filesystem_files=109, ob_tenant_server_avg_memory_used_percentage=23, process_start_time_seconds=17, go_memstats_heap_alloc_bytes=17, ob_tenant_resource_max_cpu=25, go_gc_duration_seconds_sum=17, go_memstats_mspan_inuse_bytes=17, mysql_output_write_sql_milliseconds_summary=28, ob_server_resource_memory_assigned_percent=20, monagent_pipeline_execute_seconds_total=27, go_memstats_sys_bytes=17, node_network_receive_errs_total=65, ob_all_session_num=27, observer_ulimit_max_fd_count=20, ob_plan_cache_access_total=28, ob_tenant_server_row_count=35, ob_tenant_memory_used_bytes=18, monagent_oceanbase_plan_cache_size=7, ob_tenant_server_data_size=35, go_memstats_gc_cpu_fraction=17, monagent_oceanbase_audit_cache_size=7, ob_tenant_context_memory_used_bytes=339, node_disk_writes_completed_total=39, node_load1=29, process_cpu_seconds_total=17, ob_tenant_resource_max_memory=25, go_memstats_stack_inuse_bytes=17, ob_tenant_user_partition_required_size=12, node_load5=29, monagent_input_sql_duration_seconds_count=60, node_disk_written_bytes_total=39, monagent_oceanbase_ash_cache_size=7, ob_tenant_context_memory_hold_bytes=339, go_goroutines=17, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=21, exposer_request_latencies_sum=8, ob_unit_config_min_cpu=45, process_virtual_memory_max_bytes=17, monagent_oceanbase_plan_timeinfo_cache_size=7, monagent_plugin_execute_total=41, ob_tenant500_memory_hold_bytes=20, exposer_scrapes_total=8, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=22, ob_compaction_start_time=18, oceanbase_connectivity=21} 2023-11-27 13:37:34.007 INFO 10 --- [pool-ocp-schedules-11,,1e1489887967] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:37:34.123 INFO 10 --- [http-nio-0.0.0.0-8080-exec-5,6fe0233d209044a6,b2a8cfe3d63a] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.13, traceId=6fe0233d209044a6, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:37:34.130 INFO 10 --- [http-nio-0.0.0.0-8080-exec-5,6fe0233d209044a6,b2a8cfe3d63a] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.13, traceId=6fe0233d209044a6, duration=8 ms] 2023-11-27 13:37:34.236 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:37:34.823 INFO 10 --- [alarm-detect-15,,dc869ef4e08f] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_cluster_exists_inactive_server detect time range 1701063434 ~ 1701063444 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={inactive_server_count > 0=[ExpressionEvaluateResult(matchValue=0.0, metric=inactive_server_count, match=true, value=1.0, values={1701063440=1.0}, valueCount=1, labels={app=OB, ob_cluster_id=10, ob_cluster_name=darenOB, server_ips=133.197.204.3})]}, matched=true, message=null, level=2) 2023-11-27 13:37:34.903 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:34.949 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:37:34.988 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:37:35.033 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:37:35.103 INFO 10 --- [pool-prometheus-second-schedule-1,,e2ad2e9d2aa7] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=14, collectTaskCount=14 2023-11-27 13:37:35.104 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063455, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:35.104 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063455, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:35.106 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063455, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:35.110 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063455, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:35.111 INFO 10 --- [pool-prometheus-second-schedule-1,,36047a375af9] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=11, collectTaskCount=11 2023-11-27 13:37:35.113 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063455, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:35.624 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:37:35.671 INFO 10 --- [pool-ocp-schedules-20,,e1d2392f23d5] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:37:35.671 INFO 10 --- [pool-ocp-schedules-20,,e1d2392f23d5] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:37:36.172 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:37:36.265 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:36.956 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:37:36.968 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:37:37.640 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:39.063 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:39.489 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.120.0.1:8080 2023-11-27 13:37:39.622 INFO 10 --- [http-nio-0.0.0.0-8080-exec-7,543242841db044cc,9178ca4d3736] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=133.197.206.97, traceId=543242841db044cc, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-11-27 13:37:39.625 INFO 10 --- [http-nio-0.0.0.0-8080-exec-7,543242841db044cc,9178ca4d3736] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.206.97, traceId=543242841db044cc, duration=3 ms] 2023-11-27 13:37:39.968 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:37:39.969 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:37:40.104 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063460, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:40.104 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063460, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:40.105 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063460, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:40.106 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063460, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:40.106 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063460, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:40.468 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:41.785 INFO 10 --- [pool-overview-container-4,,f8b858e17cde] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:37:41.910 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:41.977 INFO 10 --- [pool-scanner--11035-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=112 2023-11-27 13:37:42.846 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:37:42.845476+08:00, lastProcessTime=2023-11-27T13:37:32.761126+08:00 2023-11-27 13:37:42.969 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:37:42.972 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:37:43.028 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : groupAggregate, now=2023-11-27T13:37:42.845476+08:00, groupByKey=0:ob_cluster_exists_inactive_server:2, alarmsToSentCount=1, alarmIds=[44798] 2023-11-27 13:37:43.028 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : createNotification start, groupByKey=0:ob_cluster_exists_inactive_server:2 2023-11-27 13:37:43.029 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : createNotification done, groupByKey=0:ob_cluster_exists_inactive_server:2, alarmIds=[44798] 2023-11-27 13:37:43.029 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.ChannelAggregator : aggregate done, channelId=100001, now=2023-11-27T13:37:42.845476+08:00, resultsCount=1 2023-11-27 13:37:43.029 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : aggregate done, channelId=100001, alarmsCount=0, resultsCount=1 2023-11-27 13:37:43.029 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.ChannelAggregator : groupAggregate markSendSuccess, groupByKey=0:ob_cluster_exists_inactive_server:2, now=2023-11-27T13:37:42.845476+08:00, alarmIds=[44798] 2023-11-27 13:37:43.029 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:37:42.845476+08:00 2023-11-27 13:37:43.489 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:44.027 INFO 10 --- [pool-ocp-schedules-5,,7cfeae729d1d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:37:44.098 INFO 10 --- [http-nio-0.0.0.0-8080-exec-6,02ddf6f278c44a06,b4c82f0053a3] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.10, traceId=02ddf6f278c44a06, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:37:44.139 INFO 10 --- [http-nio-0.0.0.0-8080-exec-6,02ddf6f278c44a06,b4c82f0053a3] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.10, traceId=02ddf6f278c44a06, duration=41 ms] 2023-11-27 13:37:44.239 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:37:44.735 INFO 10 --- [alarm-detect-1,,a6fb7aa09430] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_cluster_exists_inactive_server detect time range 1701063444 ~ 1701063454 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={inactive_server_count > 0=[ExpressionEvaluateResult(matchValue=0.0, metric=inactive_server_count, match=true, value=1.0, values={1701063450=1.0}, valueCount=1, labels={app=OB, ob_cluster_id=10, ob_cluster_name=darenOB, server_ips=133.197.204.3})]}, matched=true, message=null, level=2) 2023-11-27 13:37:44.933 INFO 10 --- [http-nio-0.0.0.0-8080-exec-1,0562c5fe6b9c4ff6,fa411c328a9e] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.11, traceId=0562c5fe6b9c4ff6, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:37:44.969 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:37:44.998 INFO 10 --- [http-nio-0.0.0.0-8080-exec-1,0562c5fe6b9c4ff6,fa411c328a9e] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.11, traceId=0562c5fe6b9c4ff6, duration=66 ms] 2023-11-27 13:37:45.026 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:37:45.045 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:37:45.114 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063465, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:45.114 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063465, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:45.114 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063465, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:45.120 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063465, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:45.121 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063465, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:45.196 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:45.632 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:37:45.972 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:37:45.973 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:37:46.194 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:37:46.323 INFO 10 --- [http-nio-0.0.0.0-8080-exec-3,7a3af0ec6f3c492f,eeb4879ddce9] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.10, traceId=7a3af0ec6f3c492f, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:37:46.338 INFO 10 --- [http-nio-0.0.0.0-8080-exec-3,7a3af0ec6f3c492f,eeb4879ddce9] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.10, traceId=7a3af0ec6f3c492f, duration=17 ms] 2023-11-27 13:37:46.629 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:47.898 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:48.974 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:37:48.977 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:37:49.277 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:50.108 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063470, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:50.108 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063470, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:50.108 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063470, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:50.109 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063470, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:50.109 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063470, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:50.589 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:51.564 INFO 10 --- [pool-ocp-schedules-19,,fdac0efb816a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-11-27 13:37:51.595 INFO 10 --- [pool-ocp-schedules-19,,fdac0efb816a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-11-27 13:37:51.697 INFO 10 --- [pool-ocp-schedules-2,,8a4b04cebdee] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1707,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":1707,"seriesIdKeySize":87133,"secondWriteQueue":0,"minuteWriteCacheSize":32684} 2023-11-27 13:37:51.698 INFO 10 --- [pool-ocp-schedules-2,,8a4b04cebdee] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_zone_current_timestamp=38, ob_server_resource_memory_assigned_bytes=21, tcp_retrans=30, ob_unit_num=21, go_memstats_mspan_sys_bytes=17, go_memstats_stack_sys_bytes=17, exporter_export_metrics_bytes_total=28, node_filesystem_size_bytes=87, http_server_requests_seconds_count=40, ob_compaction_frozen_time=18, ob_system_event_total_waits=34, monagent_input_sql_duration_seconds_sum=60, observer_fd_count=20, ob_plan_cache_hit_total=28, processor_process_metrics_total=22, node_filesystem_readonly=86, ob_server_resource_disk_in_use_bytes=19, go_memstats_frees_total=17, sql_audit_input_collect_data_jumped_total=12, io_await=60, host_available=13, go_memstats_mallocs_total=17, mysql_output_write_sql_failed_count=23, ob_unit_config_max_memory_bytes=45, io_util=60, ob_system_event_detail_total_timeouts=782, ob_tenant_disk_v4_data_size=18, ob_latch_sleep_total=326, ob_tenant_unit_min_memory=41, ob_memstore_total_bytes=28, process_resident_memory_bytes=17, ob_cache_size_bytes=47, ob_tenant_unit_max_memory=41, ob_tenant_assigned_cpu_total=25, input_collect_metrics_total=29, ob_tenant_server_available_disk_free=28, ob_tenant_server_avg_cpu_used_percentage=23, node_disk_io_time_weighted_seconds_total=39, node_disk_write_time_seconds_total=39, cpu_count=30, output_write_metrics_total=15, ob_cluster_sync=175, process_open_fds=17, odp_transaction_total=22, ob_system_event_detail_time_waited_us=782, process_max_fds=17, ob_server_resource_cpu_assigned_percent=20, ocp_alarm_detect_duration_ms_count=82, go_memstats_heap_inuse_bytes=17, ob_tenant_server_max_cpu_used_percentage=23, proxy_memory_used_bytes=30, go_memstats_mcache_sys_bytes=17, ob_tenant_disk_v4_log_size=18, node_memory_memfree_bytes=29, process_exists=38, go_gc_duration_seconds=22, ob_server_resource_disk_bytes=21, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=27, node_memory_cached_bytes=29, ob_table_num=24, ob_disk_free_bytes=21, node_filesystem_avail_bytes=87, ob_backup_storage_capacity_bytes=349, ob_active_session_num=27, ob_unit_config_max_iops=45, ob_plan_cache_memory_bytes=28, node_network_transmit_errs_total=65, go_memstats_last_gc_time_seconds=17, go_gc_duration_seconds_count=17, monitor_exporter_avaliable=78, ob_tenant_server_max_memory_used_percentage=23, node_disk_read_bytes_total=39, mysql_output_write_sql_milliseconds_summary_count=25, ob_compaction_is_suspended=18, ob_tenant_unit_log_disk_size=22, go_memstats_lookups_total=17, mysql_output_write_sql_count=25, log_tailer_count=17, host_agent_process_status=21, ob_server_resource_cpu=21, go_memstats_alloc_bytes=17, ob_server_num=58, node_cpu_seconds_total_native=47, ob_tenant_unit_count=39, ob_session_all_num=25, ob_tenant_resource_min_memory=25, ob_memstore_freeze_times=28, ob_system_event_time_waited=34, ob_tenant_resource_min_cpu=25, ob_cluster_status_check=49, monagent_plugin_execute_seconds_total=41, node_net_bandwidth_bps=45, monagent_oceanbase_db_pool_idle=7, go_memstats_heap_idle_bytes=17, go_memstats_gc_sys_bytes=17, ob_server_resource_memory_bytes=21, ob_tenant_sys_partition_required_size=12, ob_latch_get_total=326, go_info=18, ob_tenant_assigned_mem_assigned=25, log_tailer_processed_file_count=24, node_load15=29, ob_tenant_server_required_size=35, ob_unit_config_min_iops=45, ob_server_partition_replica_limit=7, ob_tenant_unit_max_cpu=41, ob_latch_wait_seconds_total=326, node_memory_buffers_bytes=29, go_memstats_heap_released_bytes=17, node_network_receive_bytes_total=65, ob_memstore_freeze_trigger_bytes=28, ob_waitevent_wait_seconds_total=28, ob_server_stopped_duration_seconds=20, exposer_request_latencies_count=8, ob_role_is_rootserver=20, go_memstats_next_gc_bytes=17, mysql_output_context_timeout_discard_metrics_count=17, ob_session_active_num=25, node_filesystem_files_free=109, ob_memtable_max_snapshot_duration_seconds=27, node_disk_read_time_seconds_total=39, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=22, process_virtual_memory_bytes=17, mysql_output_write_sql_retry_count=23, ob_tenant_assigned_cpu_assigned=25, ob_tenant_unit_min_cpu=41, ob_unit_config_max_cpu=45, ob_unit_config_max_session_num=18, http_request_milliseconds_summary_count=79, monagent_oceanbase_audit_timeinfo_cache_size=7, net_connectivity=10, monagent_input_sql_duration_seconds_bucket=70, ob_latch_immediate_get_total=326, node_cpu_seconds_total=37, http_request_milliseconds_summary=82, ob_latch_immediate_miss_total=326, ob_index_error_num=16, ob_tenant_server_available_disk_total=28, ob_latch_spin_get_total=326, ob_tenant_memory_hold_bytes=18, go_memstats_buck_hash_sys_bytes=17, monagent_pipeline_report_metrics_total=18, mysql_output_write_sql_milliseconds_summary_sum=25, ob_tenant_disk_total_size=25, exposer_transferred_bytes_total=8, odp_sql_request_total=67, mysql_output_write_task_size=24, ob_tenant_context_memory_count=339, node_disk_reads_completed_total=39, exporter_export_metrics_total=28, ob_sysstat=129, ob_partition_num=13, node_ntp_offset_seconds=29, ob_latch_miss_total=326, go_memstats_heap_sys_bytes=17, ocp_meta_db_health=4, ob_disk_total_bytes=21, ob_zone_stat=38, go_memstats_heap_objects=17, go_memstats_mcache_inuse_bytes=17, go_memstats_other_sys_bytes=17, go_threads=17, http_request_milliseconds_summary_sum=79, ob_memstore_active_bytes=28, http_server_requests_seconds_sum=40, go_memstats_alloc_bytes_total=17, ob_tenant_server_partition_replica_count=35, ob_tenant_assigned_mem_total=25, node_memory_memtotal_bytes=29, node_network_transmit_drop_total=65, ob_system_event_detail_total_waits=782, odp_used_connections=28, ob_tenant_task_max_duration_seconds=25, backup_storage_capacity_used_percent=14, exposer_request_latencies=11, ob_waitevent_wait_total=28, odp_sql_cost_total=66, ob_tenant500_memory_used_bytes=20, node_network_transmit_bytes_total=65, ob_compaction_is_error=18, node_network_receive_drop_total=65, ob_server_partition_replica_count=7, monagent_pipeline_execute_total=27, node_filesystem_files=109, ob_tenant_server_avg_memory_used_percentage=23, process_start_time_seconds=17, go_memstats_heap_alloc_bytes=17, ob_tenant_resource_max_cpu=25, go_gc_duration_seconds_sum=17, go_memstats_mspan_inuse_bytes=17, mysql_output_write_sql_milliseconds_summary=28, ob_server_resource_memory_assigned_percent=20, monagent_pipeline_execute_seconds_total=27, go_memstats_sys_bytes=17, node_network_receive_errs_total=65, ob_all_session_num=27, observer_ulimit_max_fd_count=20, ob_plan_cache_access_total=28, ob_tenant_server_row_count=35, ob_tenant_memory_used_bytes=18, monagent_oceanbase_plan_cache_size=7, ob_tenant_server_data_size=35, go_memstats_gc_cpu_fraction=17, monagent_oceanbase_audit_cache_size=7, ob_tenant_context_memory_used_bytes=339, node_disk_writes_completed_total=39, node_load1=29, process_cpu_seconds_total=17, ob_tenant_resource_max_memory=25, go_memstats_stack_inuse_bytes=17, ob_tenant_user_partition_required_size=12, node_load5=29, monagent_input_sql_duration_seconds_count=60, node_disk_written_bytes_total=39, monagent_oceanbase_ash_cache_size=7, ob_tenant_context_memory_hold_bytes=339, go_goroutines=17, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=21, exposer_request_latencies_sum=8, ob_unit_config_min_cpu=45, process_virtual_memory_max_bytes=17, monagent_oceanbase_plan_timeinfo_cache_size=7, monagent_plugin_execute_total=41, ob_tenant500_memory_hold_bytes=20, exposer_scrapes_total=8, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=22, ob_compaction_start_time=18, oceanbase_connectivity=21} 2023-11-27 13:37:51.792 INFO 10 --- [pool-overview-container-1,,17e39340eb66] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:37:51.925 INFO 10 --- [pool-scanner--11128-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=112 2023-11-27 13:37:51.979 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:37:51.980 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:37:52.008 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:52.203 INFO 10 --- [pool-ocp-schedules-18,,b916cbac2a4f] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-11-27 13:37:52.211 INFO 10 --- [pool-ocp-schedules-10,,d840f9d82e4a] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-11-27 13:37:52.212 INFO 10 --- [pool-ocp-schedules-8,,6d73973b8bec] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-11-27 13:37:52.218 INFO 10 --- [pool-ocp-schedules-18,,b916cbac2a4f] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-11-27 13:37:52.221 INFO 10 --- [pool-ocp-schedules-8,,6d73973b8bec] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-11-27 13:37:52.222 INFO 10 --- [pool-ocp-schedules-10,,d840f9d82e4a] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-11-27 13:37:52.302 INFO 10 --- [ForkJoinPool-25-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:52.303 INFO 10 --- [ForkJoinPool-25-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:52.303 INFO 10 --- [ForkJoinPool-25-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:52.307 INFO 10 --- [ForkJoinPool-25-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:52.311 INFO 10 --- [ForkJoinPool-25-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:52.311 INFO 10 --- [ForkJoinPool-25-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:52.314 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:52.315 INFO 10 --- [ForkJoinPool-25-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:52.318 INFO 10 --- [ForkJoinPool-25-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-11-27 13:37:52.319 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:52.319 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:52.324 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:52.332 INFO 10 --- [ForkJoinPool-25-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:52.335 INFO 10 --- [ForkJoinPool-25-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:52.335 INFO 10 --- [ForkJoinPool-25-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:52.340 INFO 10 --- [ForkJoinPool-25-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:52.340 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:52.340 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:52.345 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, MAX(CASE `name` WHEN 'region' THEN `info` ELSE '' END ) `region`, MAX(CASE `name` WHEN 'idc' THEN `info` ELSE '' END ) `idc`, MAX(CASE `name` WHEN 'status' THEN `info` ELSE '' END ) `status`, MAX(CASE `name` WHEN 'merge_status' THEN `info` ELSE '' END ) `merge_status`, MAX(CASE `name` WHEN 'broadcast_version' THEN `value` ELSE 0 END ) `broadcast_version`, MAX(CASE `name` WHEN 'all_merged_version' THEN `value` ELSE 0 END ) `all_merged_version`, MAX(CASE `name` WHEN 'last_merged_version' THEN `value` ELSE 0 END ) `last_merged_version`, MAX(CASE `name` WHEN 'merge_start_time' THEN `value` ELSE 0 END ) `merge_start_time`, MAX(CASE `name` WHEN 'last_merged_time' THEN `value` ELSE 0 END ) `last_merged_time`, MAX(CASE `name` WHEN 'is_merge_timeout' THEN `value` ELSE 0 END ) `merge_timeout` FROM oceanbase.__all_zone WHERE `zone` <> '' GROUP BY `zone` 2023-11-27 13:37:52.346 INFO 10 --- [ForkJoinPool-25-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:52.348 INFO 10 --- [ForkJoinPool-25-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:52.348 INFO 10 --- [ForkJoinPool-25-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:52.349 INFO 10 --- [ForkJoinPool-25-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=3.1.4 2023-11-27 13:37:52.379 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:52.381 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:52.381 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:52.382 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:52.386 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:52.386 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:52.387 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-11-27 13:37:52.524 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:37:52.525 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:52.525 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:52.525 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:52.526 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:52.527 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:52.527 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:52.528 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, MAX(CASE `name` WHEN 'region' THEN `info` ELSE '' END ) `region`, MAX(CASE `name` WHEN 'idc' THEN `info` ELSE '' END ) `idc`, MAX(CASE `name` WHEN 'status' THEN `info` ELSE '' END ) `status`, MAX(CASE `name` WHEN 'merge_status' THEN `info` ELSE '' END ) `merge_status`, MAX(CASE `name` WHEN 'broadcast_version' THEN `value` ELSE 0 END ) `broadcast_version`, MAX(CASE `name` WHEN 'all_merged_version' THEN `value` ELSE 0 END ) `all_merged_version`, MAX(CASE `name` WHEN 'last_merged_version' THEN `value` ELSE 0 END ) `last_merged_version`, MAX(CASE `name` WHEN 'merge_start_time' THEN `value` ELSE 0 END ) `merge_start_time`, MAX(CASE `name` WHEN 'last_merged_time' THEN `value` ELSE 0 END ) `last_merged_time`, MAX(CASE `name` WHEN 'is_merge_timeout' THEN `value` ELSE 0 END ) `merge_timeout` FROM oceanbase.__all_zone WHERE `zone` <> '' GROUP BY `zone` 2023-11-27 13:37:52.530 WARN 10 --- [ForkJoinPool-26-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-11-27 13:37:52.531 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:52.531 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:52.532 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, stop_time, start_service_time, block_migrate_in_time, id FROM oceanbase.__all_server ORDER BY zone ASC, id ASC 2023-11-27 13:37:52.569 INFO 10 --- [ForkJoinPool-26-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.206.97:2882, status=ACTIVE, startServiceTime=1701063018577428, active=true 2023-11-27 13:37:52.569 INFO 10 --- [ForkJoinPool-26-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=133.197.206.97:2882, stopTime=0, stopped=false 2023-11-27 13:37:52.569 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=IDLE, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=IDLE, startTime=null, stopTime=null} 2023-11-27 13:37:52.591 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=IDLE, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:37:52.592 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=133.197.206.97, port=2882, sqlPort=2881, version=3.1.4_10000092022071511-b4bfa011ceaef428782dcb65ae89190c40b78c2f(Jul 15 2022 11:45:14), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-27T13:30:19+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.206.97, port=2882, sqlPort=2881, version=3.1.4_10000092022071511-b4bfa011ceaef428782dcb65ae89190c40b78c2f(Jul 15 2022 11:45:14), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-27T05:30:18.577428Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:37:52.718 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=133.197.206.97, port=2882, sqlPort=2881, version=3.1.4_10000092022071511-b4bfa011ceaef428782dcb65ae89190c40b78c2f(Jul 15 2022 11:45:14), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-27T05:30:18.577428Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:37:52.739 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:37:52.739 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:52.740 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:52.740 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:52.741 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:52.742 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:52.742 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:52.742 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:52.767 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:37:52.769 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:52.771 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:52.772 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:52.777 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:52.783 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:52.784 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:52.784 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:52.786 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:52.786 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:52.786 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT cluster_id, cluster_name, created, cluster_role, cluster_status, current_scn, standby_became_primary_scn, primary_cluster_id, (time_to_usec(now(6)) - current_scn)/1000000 as delay_time, protection_mode, redo_transport_options, protection_level FROM oceanbase.v$ob_cluster 2023-11-27 13:37:52.787 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:52.787 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:52.788 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT switchover_status, switchover_info FROM oceanbase.v$ob_cluster 2023-11-27 13:37:52.796 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=obcluster, obClusterId=1 2023-11-27 13:37:52.796 INFO 10 --- [ForkJoinPool-26-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"133.197.206.97:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1700793672036000}, createTime=2022-11-29T14:52:53+08:00, creator=admin, obVersion=3.1.4, tenantCount=3) 2023-11-27 13:37:52.888 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:37:52.886785+08:00, lastProcessTime=2023-11-27T13:37:42.845476+08:00 2023-11-27 13:37:53.042 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : groupAggregate, now=2023-11-27T13:37:52.886785+08:00, groupByKey=0:ob_cluster_exists_inactive_server:2, alarmsToSentCount=1, alarmIds=[44798] 2023-11-27 13:37:53.042 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : createNotification start, groupByKey=0:ob_cluster_exists_inactive_server:2 2023-11-27 13:37:53.042 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : createNotification done, groupByKey=0:ob_cluster_exists_inactive_server:2, alarmIds=[44798] 2023-11-27 13:37:53.042 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.ChannelAggregator : aggregate done, channelId=100000, now=2023-11-27T13:37:52.886785+08:00, resultsCount=1 2023-11-27 13:37:53.042 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : aggregate done, channelId=100000, alarmsCount=0, resultsCount=1 2023-11-27 13:37:53.042 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.ChannelAggregator : groupAggregate markSendSuccess, groupByKey=0:ob_cluster_exists_inactive_server:2, now=2023-11-27T13:37:52.886785+08:00, alarmIds=[44798] 2023-11-27 13:37:53.042 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:37:52.886785+08:00 2023-11-27 13:37:53.285 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:37:53.286 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:53.287 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:53.287 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:53.288 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:53.292 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:53.292 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:53.293 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-11-27 13:37:53.296 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:53.296 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:53.297 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-11-27 13:37:53.300 INFO 10 --- [ForkJoinPool-26-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.204.3:2882, status=INACTIVE, startServiceTime=0, active=false 2023-11-27 13:37:53.300 INFO 10 --- [ForkJoinPool-26-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.204.10:2882, status=ACTIVE, startServiceTime=1695730092226557, active=true 2023-11-27 13:37:53.300 INFO 10 --- [ForkJoinPool-26-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=133.197.204.10:2882, stopTime=0, stopped=false 2023-11-27 13:37:53.300 INFO 10 --- [ForkJoinPool-26-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.204.11:2882, status=ACTIVE, startServiceTime=1695730305487530, active=true 2023-11-27 13:37:53.300 INFO 10 --- [ForkJoinPool-26-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=133.197.204.11:2882, stopTime=0, stopped=false 2023-11-27 13:37:53.300 INFO 10 --- [ForkJoinPool-26-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.204.12:2882, status=ACTIVE, startServiceTime=1695730736949667, active=true 2023-11-27 13:37:53.300 INFO 10 --- [ForkJoinPool-26-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=133.197.204.12:2882, stopTime=0, stopped=false 2023-11-27 13:37:53.300 INFO 10 --- [ForkJoinPool-26-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.204.13:2882, status=ACTIVE, startServiceTime=1695731005656813, active=true 2023-11-27 13:37:53.300 INFO 10 --- [ForkJoinPool-26-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=133.197.204.13:2882, stopTime=0, stopped=false 2023-11-27 13:37:53.301 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=14, name=zone6, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone6, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-27 13:37:53.411 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.120.0.1:8080 2023-11-27 13:37:53.428 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=14, name=zone6, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:37:53.428 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=14, ip=133.197.204.13, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=7, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T20:23:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.204.13, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=7, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:23:25.656813Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:37:53.487 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=14, ip=133.197.204.13, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=7, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:23:25.656813Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:37:53.487 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=13, name=zone5, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone5, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-27 13:37:53.536 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=13, name=zone5, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:37:53.536 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=13, ip=133.197.204.12, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=6, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T20:18:57+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.204.12, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=6, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:18:56.949667Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:37:53.623 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=13, ip=133.197.204.12, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=6, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:18:56.949667Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:37:53.623 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=12, name=zone4, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone4, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-27 13:37:53.628 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.120.0.1:8080 2023-11-27 13:37:53.687 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=12, name=zone4, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:37:53.687 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=12, ip=133.197.204.11, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T20:11:45+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.204.11, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:11:45.487530Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:37:53.724 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=12, ip=133.197.204.11, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:11:45.487530Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:37:53.724 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=11, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-27 13:37:53.766 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=11, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:37:53.766 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=11, ip=133.197.204.10, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=true, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T20:08:12+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.204.10, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=true, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:08:12.226557Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:37:53.870 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.120.0.1:8080 2023-11-27 13:37:53.879 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=11, ip=133.197.204.10, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=true, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:08:12.226557Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:37:53.879 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-27 13:37:53.939 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:37:53.940 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=133.197.204.3, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=2, status=UNAVAILABLE, innerStatus=INACTIVE, startTime=1970-01-01T08:00+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.204.3, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=2, status=null, innerStatus=INACTIVE, startTime=1970-01-01T00:00Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:37:53.978 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=133.197.204.3, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=2, status=UNAVAILABLE, innerStatus=INACTIVE, startTime=1970-01-01T00:00Z, stopTime=1970-01-01T00:00Z}, oldStatus=UNAVAILABLE, newStatus=UNAVAILABLE 2023-11-27 13:37:54.047 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:37:54.048 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:54.048 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:54.048 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:54.049 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:54.052 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:54.052 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:54.053 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:54.059 INFO 10 --- [pool-ocp-schedules-11,,679076f78c06] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:37:54.085 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=darenOB, obClusterId=10 2023-11-27 13:37:54.085 INFO 10 --- [ForkJoinPool-26-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=darenOB, obClusterId=10, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"darenOB","ObClusterId":10,"ObRegion":"darenOB","ObRegionId":10,"ReadonlyRsList":[],"RsList":[{"address":"133.197.204.10:2882","role":"LEADER","sql_port":2881},{"address":"133.197.204.11:2882","role":"FOLLOWER","sql_port":2881},{"address":"133.197.204.12:2882","role":"FOLLOWER","sql_port":2881},{"address":"133.197.204.13:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1701063392653000}, createTime=2022-11-29T15:05:37+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-11-27 13:37:54.204 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:54.235 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:37:54.492 INFO 10 --- [pool-scanner--11166-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=112 2023-11-27 13:37:54.518 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.120.0.1:8080 2023-11-27 13:37:54.627 INFO 10 --- [alarm-detect-5,,e682386e1f76] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_cluster_exists_inactive_server detect time range 1701063454 ~ 1701063464 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={inactive_server_count > 0=[ExpressionEvaluateResult(matchValue=0.0, metric=inactive_server_count, match=true, value=1.0, values={1701063460=1.0}, valueCount=1, labels={app=OB, ob_cluster_id=10, ob_cluster_name=darenOB, server_ips=133.197.204.3})]}, matched=true, message=null, level=2) 2023-11-27 13:37:54.707 INFO 10 --- [pool-scanner--11222-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=112 2023-11-27 13:37:54.773 INFO 10 --- [pool-scanner--11223-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=112 2023-11-27 13:37:54.845 INFO 10 --- [pool-ocp-schedules-18,,1390c366ca3d] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:37:54.888 INFO 10 --- [pool-ocp-schedules-18,,1390c366ca3d] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:37:54.982 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:37:54.986 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:37:55.026 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:37:55.044 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:37:55.094 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:37:55.109 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063475, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:55.110 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063475, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:55.112 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063475, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:55.112 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063475, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:55.113 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063475, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:37:55.163 INFO 10 --- [pool-ocp-schedules-18,,1390c366ca3d] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:37:55.201 INFO 10 --- [pool-ocp-schedules-18,,1390c366ca3d] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:37:55.269 INFO 10 --- [pool-ocp-schedules-18,,1390c366ca3d] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:37:55.303 INFO 10 --- [pool-ocp-schedules-18,,1390c366ca3d] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:37:55.654 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:37:55.711 INFO 10 --- [pool-ocp-schedules-13,,e455988bd7bb] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:37:55.711 INFO 10 --- [pool-ocp-schedules-13,,e455988bd7bb] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:37:55.882 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:56.206 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:37:57.316 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:57.986 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:37:57.987 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:37:58.767 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:37:58.842 INFO 10 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2023-11-27 13:37:58.974 INFO 10 --- [sql-rollup-daemon,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:37:58.976 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:58.976 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:58.977 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:58.977 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:58.981 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:37:58.981 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:58.982 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, stop_time, start_service_time, block_migrate_in_time, id FROM oceanbase.__all_server ORDER BY zone ASC, id ASC 2023-11-27 13:37:59.046 INFO 10 --- [sql-rollup-daemon,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:37:59.048 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:59.049 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:59.049 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:59.050 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:37:59.053 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:37:59.054 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:59.054 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-11-27 13:37:59.056 INFO 10 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (darenOB:10-3) 2023-11-27 13:37:59.056 INFO 10 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-1) 2023-11-27 13:37:59.057 INFO 10 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (darenOB:10-4) 2023-11-27 13:37:59.057 INFO 10 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (darenOB:10-1) 2023-11-27 13:37:59.057 INFO 10 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (darenOB:10-5) 2023-11-27 13:37:59.058 INFO 10 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (darenOB:10-6) 2023-11-27 13:37:59.094 INFO 10 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:59.094 INFO 10 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:59.097 INFO 10 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:37:59.097 INFO 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=1, level=0->1, c=0) - not enough cached interval (source=2023-04-25T14:52:31, target=2023-04-25T14:50:31, granule=120000ms, delay=120000ms) 2023-11-27 13:37:59.097 INFO 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=1, level=1->2, c=0) - not enough cached interval (source=2023-04-25T14:50:31, target=2023-04-25T14:46:01, granule=600000ms, delay=270000ms) 2023-11-27 13:37:59.097 INFO 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=darenOB:10, server=1, level=0->1, c=0) - not enough cached interval (source=2023-11-24T10:14:01, target=2023-11-24T10:11:53, granule=120000ms, delay=128000ms) 2023-11-27 13:37:59.097 INFO 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=darenOB:10, server=1, level=1->2, c=0) - not enough cached interval (source=2023-11-24T10:11:53, target=2023-11-24T10:05:53, granule=600000ms, delay=360000ms) 2023-11-27 13:37:59.104 INFO 10 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:59.105 INFO 10 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:59.105 INFO 10 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:37:59.106 INFO 10 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:59.106 INFO 10 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:59.107 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=0) - no updates (start=2023-11-26T15:41:00, end=2023-11-26T15:43:30) 2023-11-27 13:37:59.107 INFO 10 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:37:59.107 INFO 10 --- [sql-rollup-worker-4,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:37:59.108 INFO 10 --- [sql-rollup-worker-4,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:37:59.109 INFO 10 --- [sql-rollup-worker-4,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:37:59.112 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=0) - no updates (start=2023-11-26T15:41:00, end=2023-11-26T15:43:30) 2023-11-27 13:37:59.115 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=0) - no updates (start=2023-11-26T15:41:00, end=2023-11-26T15:43:30) 2023-11-27 13:37:59.117 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=0) - no updates (start=2023-11-26T15:41:00, end=2023-11-26T15:43:30) 2023-11-27 13:37:59.119 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=1) - no updates (start=2023-11-26T15:43:30, end=2023-11-26T15:46:00) 2023-11-27 13:37:59.126 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=1) - no updates (start=2023-11-26T15:43:30, end=2023-11-26T15:46:00) 2023-11-27 13:37:59.131 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=1) - no updates (start=2023-11-26T15:43:30, end=2023-11-26T15:46:00) 2023-11-27 13:37:59.132 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=1) - no updates (start=2023-11-26T15:43:30, end=2023-11-26T15:46:00) 2023-11-27 13:37:59.132 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=2) - no updates (start=2023-11-26T15:46:00, end=2023-11-26T15:48:30) 2023-11-27 13:37:59.135 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=2) - no updates (start=2023-11-26T15:46:00, end=2023-11-26T15:48:30) 2023-11-27 13:37:59.136 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=2) - no updates (start=2023-11-26T15:46:00, end=2023-11-26T15:48:30) 2023-11-27 13:37:59.141 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=3) - no updates (start=2023-11-26T15:48:30, end=2023-11-26T15:51:00) 2023-11-27 13:37:59.145 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=2) - no updates (start=2023-11-26T15:46:00, end=2023-11-26T15:48:30) 2023-11-27 13:37:59.145 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=3) - no updates (start=2023-11-26T15:48:30, end=2023-11-26T15:51:00) 2023-11-27 13:37:59.146 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=3) - no updates (start=2023-11-26T15:48:30, end=2023-11-26T15:51:00) 2023-11-27 13:37:59.146 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=4) - no updates (start=2023-11-26T15:51:00, end=2023-11-26T15:53:30) 2023-11-27 13:37:59.150 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=3) - no updates (start=2023-11-26T15:48:30, end=2023-11-26T15:51:00) 2023-11-27 13:37:59.150 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=4) - no updates (start=2023-11-26T15:51:00, end=2023-11-26T15:53:30) 2023-11-27 13:37:59.153 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=4) - no updates (start=2023-11-26T15:51:00, end=2023-11-26T15:53:30) 2023-11-27 13:37:59.155 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=5) - no updates (start=2023-11-26T15:53:30, end=2023-11-26T15:56:00) 2023-11-27 13:37:59.158 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=5) - no updates (start=2023-11-26T15:53:30, end=2023-11-26T15:56:00) 2023-11-27 13:37:59.162 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=6) - no updates (start=2023-11-26T15:56:00, end=2023-11-26T15:58:30) 2023-11-27 13:37:59.165 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=7) - no updates (start=2023-11-26T15:58:30, end=2023-11-26T16:01:00) 2023-11-27 13:37:59.166 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=4) - no updates (start=2023-11-26T15:51:00, end=2023-11-26T15:53:30) 2023-11-27 13:37:59.167 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=5) - no updates (start=2023-11-26T15:53:30, end=2023-11-26T15:56:00) 2023-11-27 13:37:59.171 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=6) - no updates (start=2023-11-26T15:56:00, end=2023-11-26T15:58:30) 2023-11-27 13:37:59.173 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=8) - no updates (start=2023-11-26T16:01:00, end=2023-11-26T16:03:30) 2023-11-27 13:37:59.176 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=7) - no updates (start=2023-11-26T15:58:30, end=2023-11-26T16:01:00) 2023-11-27 13:37:59.177 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=5) - no updates (start=2023-11-26T15:53:30, end=2023-11-26T15:56:00) 2023-11-27 13:37:59.181 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=6) - no updates (start=2023-11-26T15:56:00, end=2023-11-26T15:58:30) 2023-11-27 13:37:59.181 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=8) - no updates (start=2023-11-26T16:01:00, end=2023-11-26T16:03:30) 2023-11-27 13:37:59.181 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=9) - no updates (start=2023-11-26T16:03:30, end=2023-11-26T16:06:00) 2023-11-27 13:37:59.184 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=6) - no updates (start=2023-11-26T15:56:00, end=2023-11-26T15:58:30) 2023-11-27 13:37:59.188 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=9) - no updates (start=2023-11-26T16:03:30, end=2023-11-26T16:06:00) 2023-11-27 13:37:59.189 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=7) - no updates (start=2023-11-26T15:58:30, end=2023-11-26T16:01:00) 2023-11-27 13:37:59.189 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=7) - no updates (start=2023-11-26T15:58:30, end=2023-11-26T16:01:00) 2023-11-27 13:37:59.192 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=10) - no updates (start=2023-11-26T16:06:00, end=2023-11-26T16:08:30) 2023-11-27 13:37:59.193 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=8) - no updates (start=2023-11-26T16:01:00, end=2023-11-26T16:03:30) 2023-11-27 13:37:59.194 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=10) - no updates (start=2023-11-26T16:06:00, end=2023-11-26T16:08:30) 2023-11-27 13:37:59.198 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=9) - no updates (start=2023-11-26T16:03:30, end=2023-11-26T16:06:00) 2023-11-27 13:37:59.199 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=11) - no updates (start=2023-11-26T16:08:30, end=2023-11-26T16:11:00) 2023-11-27 13:37:59.202 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=11) - no updates (start=2023-11-26T16:08:30, end=2023-11-26T16:11:00) 2023-11-27 13:37:59.202 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=8) - no updates (start=2023-11-26T16:01:00, end=2023-11-26T16:03:30) 2023-11-27 13:37:59.203 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=10) - no updates (start=2023-11-26T16:06:00, end=2023-11-26T16:08:30) 2023-11-27 13:37:59.208 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=12) - no updates (start=2023-11-26T16:11:00, end=2023-11-26T16:13:30) 2023-11-27 13:37:59.209 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=12) - no updates (start=2023-11-26T16:11:00, end=2023-11-26T16:13:30) 2023-11-27 13:37:59.210 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=9) - no updates (start=2023-11-26T16:03:30, end=2023-11-26T16:06:00) 2023-11-27 13:37:59.212 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=11) - no updates (start=2023-11-26T16:08:30, end=2023-11-26T16:11:00) 2023-11-27 13:37:59.215 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=13) - no updates (start=2023-11-26T16:13:30, end=2023-11-26T16:16:00) 2023-11-27 13:37:59.218 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=10) - no updates (start=2023-11-26T16:06:00, end=2023-11-26T16:08:30) 2023-11-27 13:37:59.218 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=13) - no updates (start=2023-11-26T16:13:30, end=2023-11-26T16:16:00) 2023-11-27 13:37:59.219 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=12) - no updates (start=2023-11-26T16:11:00, end=2023-11-26T16:13:30) 2023-11-27 13:37:59.222 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=14) - no updates (start=2023-11-26T16:16:00, end=2023-11-26T16:18:30) 2023-11-27 13:37:59.225 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=11) - no updates (start=2023-11-26T16:08:30, end=2023-11-26T16:11:00) 2023-11-27 13:37:59.225 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=14) - no updates (start=2023-11-26T16:16:00, end=2023-11-26T16:18:30) 2023-11-27 13:37:59.227 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=13) - no updates (start=2023-11-26T16:13:30, end=2023-11-26T16:16:00) 2023-11-27 13:37:59.227 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=15) - no updates (start=2023-11-26T16:18:30, end=2023-11-26T16:21:00) 2023-11-27 13:37:59.245 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=12) - no updates (start=2023-11-26T16:11:00, end=2023-11-26T16:13:30) 2023-11-27 13:37:59.245 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=14) - no updates (start=2023-11-26T16:16:00, end=2023-11-26T16:18:30) 2023-11-27 13:37:59.246 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=15) - no updates (start=2023-11-26T16:18:30, end=2023-11-26T16:21:00) 2023-11-27 13:37:59.251 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=16) - no updates (start=2023-11-26T16:21:00, end=2023-11-26T16:23:30) 2023-11-27 13:37:59.266 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=15) - no updates (start=2023-11-26T16:18:30, end=2023-11-26T16:21:00) 2023-11-27 13:37:59.266 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=16) - no updates (start=2023-11-26T16:21:00, end=2023-11-26T16:23:30) 2023-11-27 13:37:59.267 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=17) - no updates (start=2023-11-26T16:23:30, end=2023-11-26T16:26:00) 2023-11-27 13:37:59.267 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=13) - no updates (start=2023-11-26T16:13:30, end=2023-11-26T16:16:00) 2023-11-27 13:37:59.282 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=14) - no updates (start=2023-11-26T16:16:00, end=2023-11-26T16:18:30) 2023-11-27 13:37:59.294 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=16) - no updates (start=2023-11-26T16:21:00, end=2023-11-26T16:23:30) 2023-11-27 13:37:59.294 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=17) - no updates (start=2023-11-26T16:23:30, end=2023-11-26T16:26:00) 2023-11-27 13:37:59.298 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=18) - no updates (start=2023-11-26T16:26:00, end=2023-11-26T16:28:30) 2023-11-27 13:37:59.308 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=15) - no updates (start=2023-11-26T16:18:30, end=2023-11-26T16:21:00) 2023-11-27 13:37:59.311 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=18) - no updates (start=2023-11-26T16:26:00, end=2023-11-26T16:28:30) 2023-11-27 13:37:59.314 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=17) - no updates (start=2023-11-26T16:23:30, end=2023-11-26T16:26:00) 2023-11-27 13:37:59.320 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=19) - no updates (start=2023-11-26T16:28:30, end=2023-11-26T16:31:00) 2023-11-27 13:37:59.325 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=19) - no updates (start=2023-11-26T16:28:30, end=2023-11-26T16:31:00) 2023-11-27 13:37:59.327 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=16) - no updates (start=2023-11-26T16:21:00, end=2023-11-26T16:23:30) 2023-11-27 13:37:59.332 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=20) - no updates (start=2023-11-26T16:31:00, end=2023-11-26T16:33:30) 2023-11-27 13:37:59.343 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=18) - no updates (start=2023-11-26T16:26:00, end=2023-11-26T16:28:30) 2023-11-27 13:37:59.348 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=20) - no updates (start=2023-11-26T16:31:00, end=2023-11-26T16:33:30) 2023-11-27 13:37:59.349 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=17) - no updates (start=2023-11-26T16:23:30, end=2023-11-26T16:26:00) 2023-11-27 13:37:59.351 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=21) - no updates (start=2023-11-26T16:33:30, end=2023-11-26T16:36:00) 2023-11-27 13:37:59.353 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=19) - no updates (start=2023-11-26T16:28:30, end=2023-11-26T16:31:00) 2023-11-27 13:37:59.362 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=18) - no updates (start=2023-11-26T16:26:00, end=2023-11-26T16:28:30) 2023-11-27 13:37:59.365 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=21) - no updates (start=2023-11-26T16:33:30, end=2023-11-26T16:36:00) 2023-11-27 13:37:59.370 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=20) - no updates (start=2023-11-26T16:31:00, end=2023-11-26T16:33:30) 2023-11-27 13:37:59.371 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=22) - no updates (start=2023-11-26T16:36:00, end=2023-11-26T16:38:30) 2023-11-27 13:37:59.374 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=19) - no updates (start=2023-11-26T16:28:30, end=2023-11-26T16:31:00) 2023-11-27 13:37:59.380 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=23) - no updates (start=2023-11-26T16:38:30, end=2023-11-26T16:41:00) 2023-11-27 13:37:59.380 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=22) - no updates (start=2023-11-26T16:36:00, end=2023-11-26T16:38:30) 2023-11-27 13:37:59.381 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=21) - no updates (start=2023-11-26T16:33:30, end=2023-11-26T16:36:00) 2023-11-27 13:37:59.386 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=20) - no updates (start=2023-11-26T16:31:00, end=2023-11-26T16:33:30) 2023-11-27 13:37:59.387 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=24) - no updates (start=2023-11-26T16:41:00, end=2023-11-26T16:43:30) 2023-11-27 13:37:59.388 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=23) - no updates (start=2023-11-26T16:38:30, end=2023-11-26T16:41:00) 2023-11-27 13:37:59.389 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=22) - no updates (start=2023-11-26T16:36:00, end=2023-11-26T16:38:30) 2023-11-27 13:37:59.394 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=21) - no updates (start=2023-11-26T16:33:30, end=2023-11-26T16:36:00) 2023-11-27 13:37:59.395 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=23) - no updates (start=2023-11-26T16:38:30, end=2023-11-26T16:41:00) 2023-11-27 13:37:59.396 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=24) - no updates (start=2023-11-26T16:41:00, end=2023-11-26T16:43:30) 2023-11-27 13:37:59.400 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=25) - no updates (start=2023-11-26T16:43:30, end=2023-11-26T16:46:00) 2023-11-27 13:37:59.405 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=26) - no updates (start=2023-11-26T16:46:00, end=2023-11-26T16:48:30) 2023-11-27 13:37:59.408 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=24) - no updates (start=2023-11-26T16:41:00, end=2023-11-26T16:43:30) 2023-11-27 13:37:59.410 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=27) - no updates (start=2023-11-26T16:48:30, end=2023-11-26T16:51:00) 2023-11-27 13:37:59.410 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=22) - no updates (start=2023-11-26T16:36:00, end=2023-11-26T16:38:30) 2023-11-27 13:37:59.411 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=25) - no updates (start=2023-11-26T16:43:30, end=2023-11-26T16:46:00) 2023-11-27 13:37:59.413 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=25) - no updates (start=2023-11-26T16:43:30, end=2023-11-26T16:46:00) 2023-11-27 13:37:59.416 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=28) - no updates (start=2023-11-26T16:51:00, end=2023-11-26T16:53:30) 2023-11-27 13:37:59.417 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=26) - no updates (start=2023-11-26T16:46:00, end=2023-11-26T16:48:30) 2023-11-27 13:37:59.423 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=29) - no updates (start=2023-11-26T16:53:30, end=2023-11-26T16:56:00) 2023-11-27 13:37:59.425 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=23) - no updates (start=2023-11-26T16:38:30, end=2023-11-26T16:41:00) 2023-11-27 13:37:59.425 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=26) - no updates (start=2023-11-26T16:46:00, end=2023-11-26T16:48:30) 2023-11-27 13:37:59.426 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=27) - no updates (start=2023-11-26T16:48:30, end=2023-11-26T16:51:00) 2023-11-27 13:37:59.429 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=30) - no updates (start=2023-11-26T16:56:00, end=2023-11-26T16:58:30) 2023-11-27 13:37:59.435 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=28) - no updates (start=2023-11-26T16:51:00, end=2023-11-26T16:53:30) 2023-11-27 13:37:59.439 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=27) - no updates (start=2023-11-26T16:48:30, end=2023-11-26T16:51:00) 2023-11-27 13:37:59.441 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=24) - no updates (start=2023-11-26T16:41:00, end=2023-11-26T16:43:30) 2023-11-27 13:37:59.441 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=31) - no updates (start=2023-11-26T16:58:30, end=2023-11-26T17:01:00) 2023-11-27 13:37:59.452 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=29) - no updates (start=2023-11-26T16:53:30, end=2023-11-26T16:56:00) 2023-11-27 13:37:59.457 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=28) - no updates (start=2023-11-26T16:51:00, end=2023-11-26T16:53:30) 2023-11-27 13:37:59.461 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=25) - no updates (start=2023-11-26T16:43:30, end=2023-11-26T16:46:00) 2023-11-27 13:37:59.462 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=32) - no updates (start=2023-11-26T17:01:00, end=2023-11-26T17:03:30) 2023-11-27 13:37:59.471 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=26) - no updates (start=2023-11-26T16:46:00, end=2023-11-26T16:48:30) 2023-11-27 13:37:59.473 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=33) - no updates (start=2023-11-26T17:03:30, end=2023-11-26T17:06:00) 2023-11-27 13:37:59.476 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=29) - no updates (start=2023-11-26T16:53:30, end=2023-11-26T16:56:00) 2023-11-27 13:37:59.477 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=30) - no updates (start=2023-11-26T16:56:00, end=2023-11-26T16:58:30) 2023-11-27 13:37:59.480 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=27) - no updates (start=2023-11-26T16:48:30, end=2023-11-26T16:51:00) 2023-11-27 13:37:59.485 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=34) - no updates (start=2023-11-26T17:06:00, end=2023-11-26T17:08:30) 2023-11-27 13:37:59.487 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=30) - no updates (start=2023-11-26T16:56:00, end=2023-11-26T16:58:30) 2023-11-27 13:37:59.487 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=31) - no updates (start=2023-11-26T16:58:30, end=2023-11-26T17:01:00) 2023-11-27 13:37:59.507 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=28) - no updates (start=2023-11-26T16:51:00, end=2023-11-26T16:53:30) 2023-11-27 13:37:59.513 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=32) - no updates (start=2023-11-26T17:01:00, end=2023-11-26T17:03:30) 2023-11-27 13:37:59.518 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=29) - no updates (start=2023-11-26T16:53:30, end=2023-11-26T16:56:00) 2023-11-27 13:37:59.524 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=35) - no updates (start=2023-11-26T17:08:30, end=2023-11-26T17:11:00) 2023-11-27 13:37:59.525 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=33) - no updates (start=2023-11-26T17:03:30, end=2023-11-26T17:06:00) 2023-11-27 13:37:59.529 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=31) - no updates (start=2023-11-26T16:58:30, end=2023-11-26T17:01:00) 2023-11-27 13:37:59.545 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=36) - no updates (start=2023-11-26T17:11:00, end=2023-11-26T17:13:30) 2023-11-27 13:37:59.546 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=30) - no updates (start=2023-11-26T16:56:00, end=2023-11-26T16:58:30) 2023-11-27 13:37:59.546 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=34) - no updates (start=2023-11-26T17:06:00, end=2023-11-26T17:08:30) 2023-11-27 13:37:59.559 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=32) - no updates (start=2023-11-26T17:01:00, end=2023-11-26T17:03:30) 2023-11-27 13:37:59.568 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=37) - no updates (start=2023-11-26T17:13:30, end=2023-11-26T17:16:00) 2023-11-27 13:37:59.568 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=31) - no updates (start=2023-11-26T16:58:30, end=2023-11-26T17:01:00) 2023-11-27 13:37:59.569 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=35) - no updates (start=2023-11-26T17:08:30, end=2023-11-26T17:11:00) 2023-11-27 13:37:59.573 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=33) - no updates (start=2023-11-26T17:03:30, end=2023-11-26T17:06:00) 2023-11-27 13:37:59.577 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=38) - no updates (start=2023-11-26T17:16:00, end=2023-11-26T17:18:30) 2023-11-27 13:37:59.581 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=34) - no updates (start=2023-11-26T17:06:00, end=2023-11-26T17:08:30) 2023-11-27 13:37:59.581 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=32) - no updates (start=2023-11-26T17:01:00, end=2023-11-26T17:03:30) 2023-11-27 13:37:59.582 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=36) - no updates (start=2023-11-26T17:11:00, end=2023-11-26T17:13:30) 2023-11-27 13:37:59.591 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=39) - no updates (start=2023-11-26T17:18:30, end=2023-11-26T17:21:00) 2023-11-27 13:37:59.596 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=35) - no updates (start=2023-11-26T17:08:30, end=2023-11-26T17:11:00) 2023-11-27 13:37:59.596 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=33) - no updates (start=2023-11-26T17:03:30, end=2023-11-26T17:06:00) 2023-11-27 13:37:59.600 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=37) - no updates (start=2023-11-26T17:13:30, end=2023-11-26T17:16:00) 2023-11-27 13:37:59.601 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=40) - no updates (start=2023-11-26T17:21:00, end=2023-11-26T17:23:30) 2023-11-27 13:37:59.607 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=36) - no updates (start=2023-11-26T17:11:00, end=2023-11-26T17:13:30) 2023-11-27 13:37:59.608 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=34) - no updates (start=2023-11-26T17:06:00, end=2023-11-26T17:08:30) 2023-11-27 13:37:59.610 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=41) - no updates (start=2023-11-26T17:23:30, end=2023-11-26T17:26:00) 2023-11-27 13:37:59.614 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=38) - no updates (start=2023-11-26T17:16:00, end=2023-11-26T17:18:30) 2023-11-27 13:37:59.619 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=37) - no updates (start=2023-11-26T17:13:30, end=2023-11-26T17:16:00) 2023-11-27 13:37:59.619 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=35) - no updates (start=2023-11-26T17:08:30, end=2023-11-26T17:11:00) 2023-11-27 13:37:59.620 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=42) - no updates (start=2023-11-26T17:26:00, end=2023-11-26T17:28:30) 2023-11-27 13:37:59.628 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=39) - no updates (start=2023-11-26T17:18:30, end=2023-11-26T17:21:00) 2023-11-27 13:37:59.631 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=38) - no updates (start=2023-11-26T17:16:00, end=2023-11-26T17:18:30) 2023-11-27 13:37:59.632 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=36) - no updates (start=2023-11-26T17:11:00, end=2023-11-26T17:13:30) 2023-11-27 13:37:59.632 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=43) - no updates (start=2023-11-26T17:28:30, end=2023-11-26T17:31:00) 2023-11-27 13:37:59.641 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=39) - no updates (start=2023-11-26T17:18:30, end=2023-11-26T17:21:00) 2023-11-27 13:37:59.641 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=37) - no updates (start=2023-11-26T17:13:30, end=2023-11-26T17:16:00) 2023-11-27 13:37:59.643 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=44) - no updates (start=2023-11-26T17:31:00, end=2023-11-26T17:33:30) 2023-11-27 13:37:59.647 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=40) - no updates (start=2023-11-26T17:21:00, end=2023-11-26T17:23:30) 2023-11-27 13:37:59.655 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=40) - no updates (start=2023-11-26T17:21:00, end=2023-11-26T17:23:30) 2023-11-27 13:37:59.655 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=38) - no updates (start=2023-11-26T17:16:00, end=2023-11-26T17:18:30) 2023-11-27 13:37:59.658 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=45) - no updates (start=2023-11-26T17:33:30, end=2023-11-26T17:36:00) 2023-11-27 13:37:59.660 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=41) - no updates (start=2023-11-26T17:23:30, end=2023-11-26T17:26:00) 2023-11-27 13:37:59.664 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=46) - no updates (start=2023-11-26T17:36:00, end=2023-11-26T17:38:30) 2023-11-27 13:37:59.667 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=41) - no updates (start=2023-11-26T17:23:30, end=2023-11-26T17:26:00) 2023-11-27 13:37:59.667 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=39) - no updates (start=2023-11-26T17:18:30, end=2023-11-26T17:21:00) 2023-11-27 13:37:59.671 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=42) - no updates (start=2023-11-26T17:26:00, end=2023-11-26T17:28:30) 2023-11-27 13:37:59.673 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=47) - no updates (start=2023-11-26T17:38:30, end=2023-11-26T17:41:00) 2023-11-27 13:37:59.676 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=42) - no updates (start=2023-11-26T17:26:00, end=2023-11-26T17:28:30) 2023-11-27 13:37:59.677 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=40) - no updates (start=2023-11-26T17:21:00, end=2023-11-26T17:23:30) 2023-11-27 13:37:59.689 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=43) - no updates (start=2023-11-26T17:28:30, end=2023-11-26T17:31:00) 2023-11-27 13:37:59.690 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=48) - no updates (start=2023-11-26T17:41:00, end=2023-11-26T17:43:30) 2023-11-27 13:37:59.697 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=43) - no updates (start=2023-11-26T17:28:30, end=2023-11-26T17:31:00) 2023-11-27 13:37:59.697 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=41) - no updates (start=2023-11-26T17:23:30, end=2023-11-26T17:26:00) 2023-11-27 13:37:59.702 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=49) - no updates (start=2023-11-26T17:43:30, end=2023-11-26T17:46:00) 2023-11-27 13:37:59.705 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=44) - no updates (start=2023-11-26T17:31:00, end=2023-11-26T17:33:30) 2023-11-27 13:37:59.706 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=42) - no updates (start=2023-11-26T17:26:00, end=2023-11-26T17:28:30) 2023-11-27 13:37:59.711 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup delay (cluster=darenOB:10, server=6, level=0->1) - 1189 min 2023-11-27 13:37:59.715 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=44) - no updates (start=2023-11-26T17:31:00, end=2023-11-26T17:33:30) 2023-11-27 13:37:59.721 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=45) - no updates (start=2023-11-26T17:33:30, end=2023-11-26T17:36:00) 2023-11-27 13:37:59.723 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=43) - no updates (start=2023-11-26T17:28:30, end=2023-11-26T17:31:00) 2023-11-27 13:37:59.727 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=0) - no updates (start=2023-11-26T15:36:00, end=2023-11-26T15:48:00) 2023-11-27 13:37:59.731 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=45) - no updates (start=2023-11-26T17:33:30, end=2023-11-26T17:36:00) 2023-11-27 13:37:59.732 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=46) - no updates (start=2023-11-26T17:36:00, end=2023-11-26T17:38:30) 2023-11-27 13:37:59.738 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=44) - no updates (start=2023-11-26T17:31:00, end=2023-11-26T17:33:30) 2023-11-27 13:37:59.739 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=1) - no updates (start=2023-11-26T15:48:00, end=2023-11-26T16:00:00) 2023-11-27 13:37:59.746 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=2) - no updates (start=2023-11-26T16:00:00, end=2023-11-26T16:12:00) 2023-11-27 13:37:59.746 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=47) - no updates (start=2023-11-26T17:38:30, end=2023-11-26T17:41:00) 2023-11-27 13:37:59.752 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=46) - no updates (start=2023-11-26T17:36:00, end=2023-11-26T17:38:30) 2023-11-27 13:37:59.755 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=45) - no updates (start=2023-11-26T17:33:30, end=2023-11-26T17:36:00) 2023-11-27 13:37:59.758 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=48) - no updates (start=2023-11-26T17:41:00, end=2023-11-26T17:43:30) 2023-11-27 13:37:59.770 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=47) - no updates (start=2023-11-26T17:38:30, end=2023-11-26T17:41:00) 2023-11-27 13:37:59.773 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=3) - no updates (start=2023-11-26T16:12:00, end=2023-11-26T16:24:00) 2023-11-27 13:37:59.774 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=49) - no updates (start=2023-11-26T17:43:30, end=2023-11-26T17:46:00) 2023-11-27 13:37:59.775 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=46) - no updates (start=2023-11-26T17:36:00, end=2023-11-26T17:38:30) 2023-11-27 13:37:59.782 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup delay (cluster=darenOB:10, server=3, level=0->1) - 1189 min 2023-11-27 13:37:59.803 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=0) - no updates (start=2023-11-26T15:36:00, end=2023-11-26T15:48:00) 2023-11-27 13:37:59.803 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=47) - no updates (start=2023-11-26T17:38:30, end=2023-11-26T17:41:00) 2023-11-27 13:37:59.805 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=48) - no updates (start=2023-11-26T17:41:00, end=2023-11-26T17:43:30) 2023-11-27 13:37:59.805 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=4) - no updates (start=2023-11-26T16:24:00, end=2023-11-26T16:36:00) 2023-11-27 13:37:59.829 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=49) - no updates (start=2023-11-26T17:43:30, end=2023-11-26T17:46:00) 2023-11-27 13:37:59.830 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=1) - no updates (start=2023-11-26T15:48:00, end=2023-11-26T16:00:00) 2023-11-27 13:37:59.832 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=48) - no updates (start=2023-11-26T17:41:00, end=2023-11-26T17:43:30) 2023-11-27 13:37:59.832 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=5) - no updates (start=2023-11-26T16:36:00, end=2023-11-26T16:48:00) 2023-11-27 13:37:59.851 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup delay (cluster=darenOB:10, server=5, level=0->1) - 1189 min 2023-11-27 13:37:59.864 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=0) - no updates (start=2023-11-26T15:36:00, end=2023-11-26T15:48:00) 2023-11-27 13:37:59.867 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=2) - no updates (start=2023-11-26T16:00:00, end=2023-11-26T16:12:00) 2023-11-27 13:37:59.867 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=6) - no updates (start=2023-11-26T16:48:00, end=2023-11-26T17:00:00) 2023-11-27 13:37:59.871 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=49) - no updates (start=2023-11-26T17:43:30, end=2023-11-26T17:46:00) 2023-11-27 13:37:59.879 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup delay (cluster=darenOB:10, server=4, level=0->1) - 1189 min 2023-11-27 13:37:59.886 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=0) - no updates (start=2023-11-26T15:36:00, end=2023-11-26T15:48:00) 2023-11-27 13:37:59.886 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=1) - no updates (start=2023-11-26T15:48:00, end=2023-11-26T16:00:00) 2023-11-27 13:37:59.887 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=3) - no updates (start=2023-11-26T16:12:00, end=2023-11-26T16:24:00) 2023-11-27 13:37:59.893 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=4) - no updates (start=2023-11-26T16:24:00, end=2023-11-26T16:36:00) 2023-11-27 13:37:59.898 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=5) - no updates (start=2023-11-26T16:36:00, end=2023-11-26T16:48:00) 2023-11-27 13:37:59.899 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=7) - no updates (start=2023-11-26T17:00:00, end=2023-11-26T17:12:00) 2023-11-27 13:37:59.904 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=8) - no updates (start=2023-11-26T17:12:00, end=2023-11-26T17:24:00) 2023-11-27 13:37:59.909 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=6) - no updates (start=2023-11-26T16:48:00, end=2023-11-26T17:00:00) 2023-11-27 13:37:59.915 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=1) - no updates (start=2023-11-26T15:48:00, end=2023-11-26T16:00:00) 2023-11-27 13:37:59.920 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=2) - no updates (start=2023-11-26T16:00:00, end=2023-11-26T16:12:00) 2023-11-27 13:37:59.923 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=2) - no updates (start=2023-11-26T16:00:00, end=2023-11-26T16:12:00) 2023-11-27 13:37:59.928 WARN 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=9) - no updates (start=2023-11-26T17:24:00, end=2023-11-26T17:36:00) 2023-11-27 13:37:59.931 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=7) - no updates (start=2023-11-26T17:00:00, end=2023-11-26T17:12:00) 2023-11-27 13:37:59.934 INFO 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=darenOB:10, server=6, level=1->2, c=10) - not enough cached interval (source=2023-11-26T17:46:00, target=2023-11-26T17:36:00, granule=600000ms, delay=600000ms) 2023-11-27 13:37:59.941 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=8) - no updates (start=2023-11-26T17:12:00, end=2023-11-26T17:24:00) 2023-11-27 13:37:59.949 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=9) - no updates (start=2023-11-26T17:24:00, end=2023-11-26T17:36:00) 2023-11-27 13:37:59.950 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=3) - no updates (start=2023-11-26T16:12:00, end=2023-11-26T16:24:00) 2023-11-27 13:37:59.950 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=3) - no updates (start=2023-11-26T16:12:00, end=2023-11-26T16:24:00) 2023-11-27 13:37:59.952 INFO 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=darenOB:10, server=3, level=1->2, c=10) - not enough cached interval (source=2023-11-26T17:46:00, target=2023-11-26T17:36:00, granule=600000ms, delay=600000ms) 2023-11-27 13:37:59.985 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=4) - no updates (start=2023-11-26T16:24:00, end=2023-11-26T16:36:00) 2023-11-27 13:37:59.992 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=4) - no updates (start=2023-11-26T16:24:00, end=2023-11-26T16:36:00) 2023-11-27 13:38:00.001 INFO 10 --- [pool-ocp-schedules-19,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-11-27 13:38:00.001 INFO 10 --- [pool-ocp-schedules-19,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-11-27 13:38:00.003 INFO 10 --- [pool-ocp-schedules-20,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-11-27 13:38:00.003 INFO 10 --- [pool-ocp-schedules-20,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1701063480 2023-11-27 13:38:00.014 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=5) - no updates (start=2023-11-26T16:36:00, end=2023-11-26T16:48:00) 2023-11-27 13:38:00.036 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=5) - no updates (start=2023-11-26T16:36:00, end=2023-11-26T16:48:00) 2023-11-27 13:38:00.046 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=6) - no updates (start=2023-11-26T16:48:00, end=2023-11-26T17:00:00) 2023-11-27 13:38:00.049 INFO 10 --- [pool-ocp-schedules-19,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-11-27 13:38:00.072 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=6) - no updates (start=2023-11-26T16:48:00, end=2023-11-26T17:00:00) 2023-11-27 13:38:00.089 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=7) - no updates (start=2023-11-26T17:00:00, end=2023-11-26T17:12:00) 2023-11-27 13:38:00.101 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=7) - no updates (start=2023-11-26T17:00:00, end=2023-11-26T17:12:00) 2023-11-27 13:38:00.117 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063480, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:00.122 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063480, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:00.128 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063480, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:00.128 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063480, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:00.129 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063480, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:00.158 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=8) - no updates (start=2023-11-26T17:12:00, end=2023-11-26T17:24:00) 2023-11-27 13:38:00.158 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=8) - no updates (start=2023-11-26T17:12:00, end=2023-11-26T17:24:00) 2023-11-27 13:38:00.166 WARN 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=9) - no updates (start=2023-11-26T17:24:00, end=2023-11-26T17:36:00) 2023-11-27 13:38:00.167 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=9) - no updates (start=2023-11-26T17:24:00, end=2023-11-26T17:36:00) 2023-11-27 13:38:00.170 INFO 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=darenOB:10, server=4, level=1->2, c=10) - not enough cached interval (source=2023-11-26T17:46:00, target=2023-11-26T17:36:00, granule=600000ms, delay=600000ms) 2023-11-27 13:38:00.170 INFO 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=darenOB:10, server=5, level=1->2, c=10) - not enough cached interval (source=2023-11-26T17:46:00, target=2023-11-26T17:36:00, granule=600000ms, delay=600000ms) 2023-11-27 13:38:00.170 INFO 10 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=1328ms, period=120000ms 2023-11-27 13:38:00.191 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:36:08.0, nextRun=2023-11-27 13:38:00.0, currentDate=Mon Nov 27 13:38:00 CST 2023 2023-11-27 13:38:00.192 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=14, name=Sync root service list, templateName=SyncRsListSchedule, scheduleRule=0 */2 * * * ?, lastRunTime=2023-11-27 13:36:08.0, nextRunTime=2023-11-27 13:38:00.0} 2023-11-27 13:38:00.258 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync root service list, nodeCount=1, taskCount=2 2023-11-27 13:38:00.258 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:38:00.258 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:38:00.258 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync root service list, taskDefinitionId=14, state=RUNNING, operation=EXECUTE} 2023-11-27 13:38:00.275 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:37:01.0, nextRun=2023-11-27 13:38:00.0, currentDate=Mon Nov 27 13:38:00 CST 2023 2023-11-27 13:38:00.275 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-27 13:37:01.0, nextRunTime=2023-11-27 13:38:00.0} 2023-11-27 13:38:00.279 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-11-27 13:38:00.279 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:38:00.279 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-11-27 13:38:00.289 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:37:02.0, nextRun=2023-11-27 13:38:00.0, currentDate=Mon Nov 27 13:38:00 CST 2023 2023-11-27 13:38:00.289 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-27 13:37:02.0, nextRunTime=2023-11-27 13:38:00.0} 2023-11-27 13:38:00.314 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-11-27 13:38:00.315 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:38:00.315 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-11-27 13:38:00.330 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:37:02.0, nextRun=2023-11-27 13:38:00.0, currentDate=Mon Nov 27 13:38:00 CST 2023 2023-11-27 13:38:00.330 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-27 13:37:02.0, nextRunTime=2023-11-27 13:38:00.0} 2023-11-27 13:38:00.345 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-11-27 13:38:00.345 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:38:00.345 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-11-27 13:38:00.358 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:37:30.0, nextRun=2023-11-27 13:38:00.0, currentDate=Mon Nov 27 13:38:00 CST 2023 2023-11-27 13:38:00.358 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-11-27 13:37:30.0, nextRunTime=2023-11-27 13:38:00.0} 2023-11-27 13:38:00.362 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-11-27 13:38:00.362 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:38:00.362 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-11-27 13:38:00.375 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:37:03.0, nextRun=2023-11-27 13:38:00.0, currentDate=Mon Nov 27 13:38:00 CST 2023 2023-11-27 13:38:00.375 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=1000002, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-27 13:37:03.0, nextRunTime=2023-11-27 13:38:00.0} 2023-11-27 13:38:00.380 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-11-27 13:38:00.380 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:38:00.380 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=1000002, state=RUNNING, operation=EXECUTE} 2023-11-27 13:38:00.387 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000048, operation:EXECUTE, state: RUNNING 2023-11-27 13:38:00.427 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000048, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=1, startTime=2023-11-27T13:38:00.384+08:00, endTime=null} 2023-11-27 13:38:00.537 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,fd4dee1b2d64] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000048, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=8000040, task_operation=execute, latest_execution_start_time=2023-11-27T13:38:00.383+08:00, sub_task_instance_id=10000048}, listMap={cluster_ids=[1, 2]}}, executor=10.120.0.1 2023-11-27 13:38:00.557 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000049, operation:EXECUTE, state: RUNNING 2023-11-27 13:38:00.606 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000049, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=0, startTime=2023-11-27T13:38:00.531+08:00, endTime=null} 2023-11-27 13:38:00.706 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000050, operation:EXECUTE, state: RUNNING 2023-11-27 13:38:00.723 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000050, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-11-27T13:38:00.694+08:00, endTime=null} 2023-11-27 13:38:00.739 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,5a272d875455] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000049, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=8000040, task_operation=execute, latest_execution_start_time=2023-11-27T13:38:00.521+08:00, sub_task_instance_id=10000049}, listMap={cluster_ids=[1, 2]}}, executor=10.120.0.1 2023-11-27 13:38:00.760 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000051, operation:EXECUTE, state: RUNNING 2023-11-27 13:38:00.770 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,fd4dee1b2d64] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:00.776 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000051, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-11-27T13:38:00.754+08:00, endTime=null} 2023-11-27 13:38:00.778 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,fd4dee1b2d64] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:00.778 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,fd4dee1b2d64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:00.779 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,fd4dee1b2d64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:00.784 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,fd4dee1b2d64] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:00.784 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,fd4dee1b2d64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:00.786 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,fd4dee1b2d64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2023-11-27 13:38:00.786 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,5a272d875455] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000049, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:38:00.805 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000052, operation:EXECUTE, state: RUNNING 2023-11-27 13:38:00.809 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000052, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-11-27T13:38:00.801+08:00, endTime=null} 2023-11-27 13:38:00.812 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,fd4dee1b2d64] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:00.812 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,fd4dee1b2d64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:00.819 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,fd4dee1b2d64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [133.197.204.10, 2882] 2023-11-27 13:38:00.824 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,fd4dee1b2d64] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:00.824 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,fd4dee1b2d64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:00.827 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,fd4dee1b2d64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [133.197.204.11, 2882] 2023-11-27 13:38:00.832 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,16b0b9d8b187] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000050, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000041, task_operation=execute, latest_execution_start_time=2023-11-27T13:38:00.693+08:00, sub_task_instance_id=10000050}, listMap={}}, executor=10.120.0.1 2023-11-27 13:38:00.833 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,fd4dee1b2d64] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:00.833 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,fd4dee1b2d64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:00.836 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,fd4dee1b2d64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [133.197.204.12, 2882] 2023-11-27 13:38:00.845 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,fd4dee1b2d64] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:00.845 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,fd4dee1b2d64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:00.848 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,fd4dee1b2d64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [133.197.204.13, 2882] 2023-11-27 13:38:00.849 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,fd4dee1b2d64] c.o.o.s.t.b.cluster.UpdateRsListTask : assembled rootserver_json={"ObCluster":"darenOB","ObClusterId":10,"ObRegion":"darenOB","ObRegionId":10,"ReadonlyRsList":[],"RsList":[{"address":"133.197.204.10:2882","role":"LEADER","sql_port":2881},{"address":"133.197.204.11:2882","role":"FOLLOWER","sql_port":2881},{"address":"133.197.204.12:2882","role":"FOLLOWER","sql_port":2881},{"address":"133.197.204.13:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1701063480849000} 2023-11-27 13:38:00.856 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,fd4dee1b2d64] c.o.o.s.t.b.cluster.UpdateRsListTask : update rootserver_json success, clusterName=darenOB, obClusterId=10 2023-11-27 13:38:00.865 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,fd4dee1b2d64] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000048, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:38:00.904 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,bb429ef90efe] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000052, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000043, task_operation=execute, latest_execution_start_time=2023-11-27T13:38:00.800+08:00, sub_task_instance_id=10000052}, listMap={}}, executor=10.120.0.1 2023-11-27 13:38:00.911 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=6 2023-11-27 13:38:00.916 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=9 2023-11-27 13:38:00.919 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-11-27 13:38:00.920 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2023-11-27 13:38:00.938 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2023-11-27 13:38:00.939 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:38:00.940 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2023-11-27 13:38:00.944 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,16b0b9d8b187] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000050, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:38:00.953 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:38:00.957 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,a8615a76476e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000051, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000042, task_operation=execute, latest_execution_start_time=2023-11-27T13:38:00.753+08:00, sub_task_instance_id=10000051}, listMap={}}, executor=10.120.0.1 2023-11-27 13:38:00.971 INFO 10 --- [pool-prometheus-minute-schedule-1,,72029a5c6622] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1701063540, interval=60 2023-11-27 13:38:00.973 INFO 10 --- [pool-prometheus-minute-schedule-2,,fbcbe0946a1b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1701063540, interval=60 2023-11-27 13:38:00.983 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,bb429ef90efe] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obcluster, sql count is 0. 2023-11-27 13:38:00.983 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,bb429ef90efe] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, result is empty. 2023-11-27 13:38:00.985 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,bb429ef90efe] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster darenOB, sql count is 0. 2023-11-27 13:38:00.985 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,bb429ef90efe] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster darenOB, result is empty. 2023-11-27 13:38:00.991 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:38:01.002 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:38:01.006 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,bb429ef90efe] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000052, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:38:01.010 INFO 10 --- [pool-prometheus-minute-schedule-2,,fbcbe0946a1b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2023-11-27 13:38:01.010 INFO 10 --- [pool-prometheus-minute-schedule-2,,fbcbe0946a1b] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1701063480, count=8, elapsed=1699362417530 2023-11-27 13:38:01.013 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62888/metrics/stat, collectAt=1701063480, message=java.net.ConnectException: Connection refused: /133.197.206.92:62888, rootCause=ConnectException: Connection refused 2023-11-27 13:38:01.013 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/stat, collectAt=1701063480, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:01.014 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/stat, collectAt=1701063480, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:01.028 INFO 10 --- [pool-prometheus-minute-schedule-1,,72029a5c6622] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=11, collectTaskCount=11 2023-11-27 13:38:01.028 INFO 10 --- [pool-prometheus-minute-schedule-1,,72029a5c6622] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1701063480, count=11, elapsed=1699362417548 2023-11-27 13:38:01.030 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62888/metrics/stat, collectAt=1701063480, message=java.net.ConnectException: Connection refused: /133.197.206.97:62888, rootCause=ConnectException: Connection refused 2023-11-27 13:38:01.032 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/extra, collectAt=1701063480, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:01.032 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62888/metrics/stat, collectAt=1701063480, message=java.net.ConnectException: Connection refused: /133.197.206.88:62888, rootCause=ConnectException: Connection refused 2023-11-27 13:38:01.034 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:38:01.040 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/stat, collectAt=1701063480, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:01.153 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.88:62889/api/v1/log/alarms, params:null 2023-11-27 13:38:01.187 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-11-27 13:38:01.199 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,a8615a76476e] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-11-27 13:38:01.203 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,a8615a76476e] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster darenOB, trans stat result is empty. 2023-11-27 13:38:01.248 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,a8615a76476e] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000051, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:38:01.260 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.12:62889/api/v1/log/alarms, params:null 2023-11-27 13:38:01.263 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.13:62889/api/v1/log/alarms, params:null 2023-11-27 13:38:01.273 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-11-27 13:38:01.278 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-11-27 13:38:01.307 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:38:01.312 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-11-27 13:38:01.313 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.92:62889/api/v1/log/alarms, params:null 2023-11-27 13:38:01.356 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:38:01.438 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000040, status:SUCCESS 2023-11-27 13:38:01.442 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000040, status:SUCCESS 2023-11-27 13:38:01.472 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.11:62889/api/v1/log/alarms, params:null 2023-11-27 13:38:01.481 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000041, status:SUCCESS 2023-11-27 13:38:01.483 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000041, status:SUCCESS 2023-11-27 13:38:01.496 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.3:62889/api/v1/log/alarms, params:null 2023-11-27 13:38:01.505 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000042, status:SUCCESS 2023-11-27 13:38:01.507 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000042, status:SUCCESS 2023-11-27 13:38:01.532 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.97:62889/api/v1/log/alarms, params:null 2023-11-27 13:38:01.553 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000043, status:SUCCESS 2023-11-27 13:38:01.556 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000043, status:SUCCESS 2023-11-27 13:38:01.790 INFO 10 --- [pool-overview-container-6,,969901308d18] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:38:01.851 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000053, operation:EXECUTE, state: RUNNING 2023-11-27 13:38:01.856 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000053, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-11-27T13:38:01.846+08:00, endTime=null} 2023-11-27 13:38:01.888 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000054, operation:EXECUTE, state: RUNNING 2023-11-27 13:38:01.897 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000054, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-11-27T13:38:01.873+08:00, endTime=null} 2023-11-27 13:38:01.906 INFO 10 --- [pool-scanner--11224-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=100 2023-11-27 13:38:01.916 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,3ef578988cfd] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000053, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000044, task_operation=execute, latest_execution_start_time=2023-11-27T13:38:01.846+08:00, sub_task_instance_id=10000053}, listMap={}}, executor=10.120.0.1 2023-11-27 13:38:01.916 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,3ef578988cfd] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-11-27 13:38:01.946 INFO 10 --- [pool-588-thread-1,554481c3fd1b426b,f53e4c4d7290] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:darenOB, tenant:sys 2023-11-27 13:38:01.947 INFO 10 --- [pool-588-thread-2,554481c3fd1b426b,ded695473870] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:darenOB, tenant:daren 2023-11-27 13:38:01.996 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9a05f38a84b8] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000054, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000045, task_operation=execute, latest_execution_start_time=2023-11-27T13:38:01.873+08:00, sub_task_instance_id=10000054}, listMap={}}, executor=10.120.0.1 2023-11-27 13:38:02.006 INFO 10 --- [pool-588-thread-1,554481c3fd1b426b,f53e4c4d7290] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:02.011 INFO 10 --- [pool-588-thread-1,554481c3fd1b426b,f53e4c4d7290] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:02.012 INFO 10 --- [pool-588-thread-1,554481c3fd1b426b,f53e4c4d7290] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:02.019 INFO 10 --- [pool-588-thread-1,554481c3fd1b426b,f53e4c4d7290] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:02.020 INFO 10 --- [pool-588-thread-2,554481c3fd1b426b,ded695473870] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:02.031 INFO 10 --- [pool-588-thread-2,554481c3fd1b426b,ded695473870] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:02.031 INFO 10 --- [pool-588-thread-2,554481c3fd1b426b,ded695473870] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:02.037 INFO 10 --- [pool-588-thread-1,554481c3fd1b426b,f53e4c4d7290] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:02.037 INFO 10 --- [pool-588-thread-1,554481c3fd1b426b,f53e4c4d7290] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:02.040 INFO 10 --- [pool-588-thread-2,554481c3fd1b426b,ded695473870] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:02.042 INFO 10 --- [pool-588-thread-1,554481c3fd1b426b,f53e4c4d7290] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-11-27 13:38:02.051 INFO 10 --- [pool-588-thread-2,554481c3fd1b426b,ded695473870] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:02.051 INFO 10 --- [pool-588-thread-2,554481c3fd1b426b,ded695473870] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:02.053 INFO 10 --- [pool-588-thread-2,554481c3fd1b426b,ded695473870] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-11-27 13:38:02.064 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9a05f38a84b8] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000054, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:38:02.090 INFO 10 --- [pool-588-thread-1,554481c3fd1b426b,f53e4c4d7290] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1701021601748952808, broadcast scn:1701021601748952808, saved frozen scn:1701021601748952808 2023-11-27 13:38:02.090 INFO 10 --- [pool-588-thread-1,554481c3fd1b426b,f53e4c4d7290] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:darenOB, tenant:sys 2023-11-27 13:38:02.091 INFO 10 --- [pool-588-thread-2,554481c3fd1b426b,ded695473870] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1701021600396785720, broadcast scn:1701021600396785720, saved frozen scn:1701021600396785720 2023-11-27 13:38:02.092 INFO 10 --- [pool-588-thread-2,554481c3fd1b426b,ded695473870] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:darenOB, tenant:daren 2023-11-27 13:38:02.097 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,3ef578988cfd] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-11-27 13:38:02.119 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,3ef578988cfd] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000053, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:38:02.701 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000044, status:SUCCESS 2023-11-27 13:38:02.702 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000044, status:SUCCESS 2023-11-27 13:38:02.727 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000045, status:SUCCESS 2023-11-27 13:38:02.729 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000045, status:SUCCESS 2023-11-27 13:38:02.832 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:38:02.831310+08:00, lastProcessTime=2023-11-27T13:37:52.886785+08:00 2023-11-27 13:38:02.933 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:38:03.154 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:38:02.831310+08:00 2023-11-27 13:38:04.002 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:38:04.003 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:38:04.122 INFO 10 --- [pool-ocp-schedules-15,,2a7b4c70716f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:38:04.142 INFO 10 --- [pool-ocp-schedules-19,,c1b2f97faee0] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=585625099342655488 2023-11-27 13:38:04.148 INFO 10 --- [pool-ocp-schedules-19,,c1b2f97faee0] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, newSeriesIdsCount=2, maxSyncedSeriesId=585625224722984960 2023-11-27 13:38:04.235 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:38:04.585 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:38:04.645 INFO 10 --- [alarm-detect-2,,35877004fa40] c.o.o.s.a.core.detect.MetricDetector : alarmType host_unavailable detect time range 1701063414 ~ 1701063474 matched, alarm-level: 1, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={host_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=host_available, match=true, value=0.0, values={1701063420=0.0}, valueCount=1, labels={app=HOST, host=133.197.206.92, host_name=localhost.localdomain, ob_cluster_id=, ob_cluster_name=, svr_ip=133.197.206.92})]}, matched=true, message=null, level=1) 2023-11-27 13:38:04.646 INFO 10 --- [alarm-detect-2,,35877004fa40] c.o.o.s.a.core.detect.MetricDetector : alarmType host_unavailable detect time range 1701063414 ~ 1701063474 matched, alarm-level: 1, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={host_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=host_available, match=true, value=0.0, values={1701063420=0.0}, valueCount=1, labels={app=HOST, host=133.197.206.97, host_name=localhost.localdomain, ob_cluster_id=1, ob_cluster_name=obcluster, svr_ip=133.197.206.97})]}, matched=true, message=null, level=1) 2023-11-27 13:38:04.646 INFO 10 --- [alarm-detect-2,,35877004fa40] c.o.o.s.a.core.detect.MetricDetector : alarmType host_unavailable detect time range 1701063414 ~ 1701063474 matched, alarm-level: 1, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={host_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=host_available, match=true, value=0.0, values={1701063420=0.0}, valueCount=1, labels={app=HOST, host=133.197.206.88, host_name=localhost.localdomain, ob_cluster_id=, ob_cluster_name=, svr_ip=133.197.206.88})]}, matched=true, message=null, level=1) 2023-11-27 13:38:04.646 INFO 10 --- [alarm-detect-2,,35877004fa40] c.o.o.s.a.core.detect.MetricDetector : alarmType host_unavailable detect time range 1701063414 ~ 1701063474 matched, alarm-level: 1, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={host_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=host_available, match=true, value=0.0, values={1701063420=0.0}, valueCount=1, labels={app=HOST, host=133.197.204.3, host_name=localhost.localdomain, ob_cluster_id=10, ob_cluster_name=darenOB, svr_ip=133.197.204.3})]}, matched=true, message=null, level=1) 2023-11-27 13:38:04.664 INFO 10 --- [alarm-detect-3,,6fce5b5e7d6a] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063414 ~ 1701063474 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063420=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/obproxy, exporter_addr=http://133.197.204.3:62889/metrics/obproxy, exporter_type=OB_PROXY, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:38:04.665 INFO 10 --- [alarm-detect-3,,6fce5b5e7d6a] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063414 ~ 1701063474 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063420=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/ob/basic, exporter_addr=http://133.197.206.97:62889/metrics/ob/basic, exporter_type=OB_CLUSTER, host=133.197.206.97, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.206.97})]}, matched=true, message=null, level=3) 2023-11-27 13:38:04.665 INFO 10 --- [alarm-detect-3,,6fce5b5e7d6a] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063414 ~ 1701063474 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063420=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/ob, exporter_addr=http://133.197.206.97:62889/metrics/node/ob, exporter_type=OB_CLUSTER, host=133.197.206.97, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.206.97})]}, matched=true, message=null, level=3) 2023-11-27 13:38:04.665 INFO 10 --- [alarm-detect-3,,6fce5b5e7d6a] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063414 ~ 1701063474 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063420=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/ob/basic, exporter_addr=http://133.197.204.3:62889/metrics/ob/basic, exporter_type=OB_CLUSTER, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:38:04.665 INFO 10 --- [alarm-detect-3,,6fce5b5e7d6a] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063414 ~ 1701063474 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063420=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://133.197.204.3:62888/metrics/stat, exporter_type=HOST_MONITOR, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:38:04.666 INFO 10 --- [alarm-detect-3,,6fce5b5e7d6a] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063414 ~ 1701063474 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063420=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/host, exporter_addr=http://133.197.204.3:62889/metrics/node/host, exporter_type=HOST_MONITOR, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:38:04.666 INFO 10 --- [alarm-detect-3,,6fce5b5e7d6a] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063414 ~ 1701063474 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063420=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://133.197.204.3:62889/metrics/stat, exporter_type=HOST_MONITOR, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:38:04.666 INFO 10 --- [alarm-detect-3,,6fce5b5e7d6a] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063414 ~ 1701063474 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063420=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/host, exporter_addr=http://133.197.206.92:62889/metrics/node/host, exporter_type=HOST_MONITOR, host=133.197.206.92, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.206.92})]}, matched=true, message=null, level=3) 2023-11-27 13:38:04.668 INFO 10 --- [alarm-detect-3,,6fce5b5e7d6a] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063414 ~ 1701063474 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063420=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/host, exporter_addr=http://133.197.206.88:62889/metrics/node/host, exporter_type=HOST_MONITOR, host=133.197.206.88, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.206.88})]}, matched=true, message=null, level=3) 2023-11-27 13:38:04.668 INFO 10 --- [alarm-detect-3,,6fce5b5e7d6a] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063414 ~ 1701063474 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063420=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/ob, exporter_addr=http://133.197.204.3:62889/metrics/node/ob, exporter_type=OB_CLUSTER, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:38:04.668 INFO 10 --- [alarm-detect-3,,6fce5b5e7d6a] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063414 ~ 1701063474 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063420=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/ob/extra, exporter_addr=http://133.197.204.3:62889/metrics/ob/extra, exporter_type=OB_CLUSTER, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:38:04.668 INFO 10 --- [alarm-detect-3,,6fce5b5e7d6a] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063414 ~ 1701063474 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063420=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/host, exporter_addr=http://133.197.206.97:62889/metrics/node/host, exporter_type=HOST_MONITOR, host=133.197.206.97, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.206.97})]}, matched=true, message=null, level=3) 2023-11-27 13:38:04.668 INFO 10 --- [alarm-detect-3,,6fce5b5e7d6a] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063414 ~ 1701063474 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063420=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/obproxy, exporter_addr=http://133.197.204.3:62889/metrics/node/obproxy, exporter_type=OB_PROXY, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:38:04.668 INFO 10 --- [alarm-detect-3,,6fce5b5e7d6a] c.o.o.s.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=monitor_exporter_unavaliable, target=alarm_template_id=0:host=133.197.206.97:exporter=/metrics/ob/basic-3, currentState=Pending 2023-11-27 13:38:04.669 INFO 10 --- [alarm-detect-3,,6fce5b5e7d6a] c.o.o.s.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=monitor_exporter_unavaliable, target=alarm_template_id=0:host=133.197.206.97:exporter=/metrics/node/ob-3, currentState=Pending 2023-11-27 13:38:04.669 INFO 10 --- [alarm-detect-3,,6fce5b5e7d6a] c.o.o.s.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=monitor_exporter_unavaliable, target=alarm_template_id=0:host=133.197.206.92:exporter=/metrics/node/host-3, currentState=Pending 2023-11-27 13:38:04.670 INFO 10 --- [alarm-detect-3,,6fce5b5e7d6a] c.o.o.s.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=monitor_exporter_unavaliable, target=alarm_template_id=0:host=133.197.206.88:exporter=/metrics/node/host-3, currentState=Pending 2023-11-27 13:38:04.670 INFO 10 --- [alarm-detect-3,,6fce5b5e7d6a] c.o.o.s.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=monitor_exporter_unavaliable, target=alarm_template_id=0:host=133.197.206.97:exporter=/metrics/node/host-3, currentState=Pending 2023-11-27 13:38:04.676 INFO 10 --- [alarm-detect-4,,a76c6c894aea] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_cluster_exists_inactive_server detect time range 1701063464 ~ 1701063474 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={inactive_server_count > 0=[ExpressionEvaluateResult(matchValue=0.0, metric=inactive_server_count, match=true, value=1.0, values={1701063470=1.0}, valueCount=1, labels={app=OB, ob_cluster_id=10, ob_cluster_name=darenOB, server_ips=133.197.204.3})]}, matched=true, message=null, level=2) 2023-11-27 13:38:05.033 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:38:05.050 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:38:05.108 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063485, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:05.109 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063485, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:05.109 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063485, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:05.109 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063485, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:05.109 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063485, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:05.131 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:38:05.659 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:38:05.969 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:37:05.0, nextRun=2023-11-27 13:38:05.0, currentDate=Mon Nov 27 13:38:05 CST 2023 2023-11-27 13:38:05.970 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-11-27 13:37:05.0, nextRunTime=2023-11-27 13:38:05.0} 2023-11-27 13:38:05.977 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-11-27 13:38:05.977 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:38:05.977 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-11-27 13:38:06.068 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000055, operation:EXECUTE, state: RUNNING 2023-11-27 13:38:06.075 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000055, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-11-27T13:38:06.064+08:00, endTime=null} 2023-11-27 13:38:06.127 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,444c411dc9b8] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000055, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000046, task_operation=execute, latest_execution_start_time=2023-11-27T13:38:06.063+08:00, sub_task_instance_id=10000055}, listMap={}}, executor=10.120.0.1 2023-11-27 13:38:06.241 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:38:06.386 INFO 10 --- [pool-Check obproxy15,554481c3fd1b426b,f2bc93c86baa] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-11-27 13:38:06.391 INFO 10 --- [pool-Check obproxy11,554481c3fd1b426b,ee6363c0887f] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-11-27 13:38:06.393 WARN 10 --- [pool-Check obproxy11,554481c3fd1b426b,ee6363c0887f] c.oceanbase.ocp.odpsdk.OdpJdbcOperator : Failed to check odp connection Connection {driverClassName ='com.oceanbase.jdbc.Driver', connectionProperties ='[user=root@proxysys, url=jdbc:oceanbase://133.197.204.3:2883/?connectProxy=true]'}, error is Failed to obtain JDBC Connection; nested exception is java.sql.SQLNonTransientConnectionException: Could not connect to HostAddress{host='133.197.204.3', port=2883}. Connection refused (Connection refused) 2023-11-27 13:38:06.393 WARN 10 --- [pool-Check obproxy11,554481c3fd1b426b,ee6363c0887f] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to connect obproxy server, maybe caused by odp process down, wrong password, or ocp address not in private ip list and odp parameter skip_proxy_sys_private_check is not true 2023-11-27 13:38:06.393 ERROR 10 --- [pool-Check obproxy11,554481c3fd1b426b,ee6363c0887f] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.UnexpectedException occurred with code error.obproxy.connection.error, and args [133.197.204.3:2883] 2023-11-27 13:38:06.393 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 1000 lines were deleted 2023-11-27 13:38:06.394 INFO 10 --- [pool-Check obproxy12,554481c3fd1b426b,e0a92455e7f3] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-11-27 13:38:06.402 INFO 10 --- [pool-Check obproxy14,554481c3fd1b426b,ee02edf39932] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-11-27 13:38:06.402 INFO 10 --- [pool-Check obproxy13,554481c3fd1b426b,512579ca76de] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-11-27 13:38:06.427 INFO 10 --- [pool-Check obproxy15,554481c3fd1b426b,f2bc93c86baa] c.o.o.s.o.obproxy.ObproxyService : check obproxy 12 success, update last active time to now 2023-11-27 13:38:06.449 WARN 10 --- [pool-Check obproxy11,554481c3fd1b426b,ee6363c0887f] c.oceanbase.ocp.odpsdk.OdpJdbcOperator : Failed to check odp connection Connection {driverClassName ='com.oceanbase.jdbc.Driver', connectionProperties ='[user=root@proxysys, url=jdbc:oceanbase://133.197.204.3:2883/?connectProxy=true]'}, error is Failed to obtain JDBC Connection; nested exception is java.sql.SQLNonTransientConnectionException: Could not connect to HostAddress{host='133.197.204.3', port=2883}. Connection refused (Connection refused) 2023-11-27 13:38:06.449 WARN 10 --- [pool-Check obproxy11,554481c3fd1b426b,ee6363c0887f] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to connect obproxy server, maybe caused by odp process down, wrong password, or ocp address not in private ip list and odp parameter skip_proxy_sys_private_check is not true 2023-11-27 13:38:06.449 ERROR 10 --- [pool-Check obproxy11,554481c3fd1b426b,ee6363c0887f] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.UnexpectedException occurred with code error.obproxy.connection.error, and args [133.197.204.3:2883] 2023-11-27 13:38:06.452 INFO 10 --- [pool-Check obproxy13,554481c3fd1b426b,512579ca76de] c.o.o.s.o.obproxy.ObproxyService : check obproxy 10 success, update last active time to now 2023-11-27 13:38:06.467 INFO 10 --- [pool-Check obproxy12,554481c3fd1b426b,e0a92455e7f3] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-11-27 13:38:06.469 INFO 10 --- [pool-Check obproxy14,554481c3fd1b426b,ee02edf39932] c.o.o.s.o.obproxy.ObproxyService : check obproxy 11 success, update last active time to now 2023-11-27 13:38:06.484 INFO 10 --- [pool-Check obproxy11,554481c3fd1b426b,ee6363c0887f] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 failed, will not update last active time com.oceanbase.ocp.core.exception.UnexpectedException: [OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=OBPROXY_CONNECTION_ERROR, args=133.197.204.3:2883 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_312] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_312] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_312] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_312] at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.util.ExceptionUtils.throwException(ExceptionUtils.java:90) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.util.ExceptionUtils.unExpected(ExceptionUtils.java:77) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.service.operation.obproxy.SystemOdpOperatorFactory.createOdpOperator(SystemOdpOperatorFactory.java:105) ~[ocp-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.service.operation.obproxy.SystemOdpOperatorFactory.tryToCreateOdpOperator(SystemOdpOperatorFactory.java:86) ~[ocp-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.service.operation.obproxy.ObproxyService.lambda$checkObproxyServers$20(ObproxyService.java:517) ~[ocp-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] 2023-11-27 13:38:06.507 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,444c411dc9b8] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000055, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:38:06.551 INFO 10 --- [http-nio-0.0.0.0-8080-exec-5,96d36fb67b2342a4,86f96d8d971c] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.11, traceId=96d36fb67b2342a4, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:38:06.579 INFO 10 --- [http-nio-0.0.0.0-8080-exec-5,96d36fb67b2342a4,86f96d8d971c] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.11, traceId=96d36fb67b2342a4, duration=29 ms] 2023-11-27 13:38:06.995 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000046, status:SUCCESS 2023-11-27 13:38:06.997 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000046, status:SUCCESS 2023-11-27 13:38:07.004 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:38:07.006 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:38:09.266 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 175 lines were deleted 2023-11-27 13:38:09.524 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.120.0.1:8080 2023-11-27 13:38:10.006 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:38:10.007 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:38:10.102 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063490, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:10.104 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063490, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:10.104 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063490, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:10.104 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063490, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:10.107 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063490, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:11.789 INFO 10 --- [pool-overview-container-2,,fffaeb099aa5] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:38:11.816 INFO 10 --- [table-cleaning,,] c.o.o.s.dataclean.TableCleaningService : delete task in progress, 0 lines were deleted 2023-11-27 13:38:11.823 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,505d93a36926] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000026, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:38:11.973 INFO 10 --- [pool-scanner--11355-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=100 2023-11-27 13:38:12.252 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000020, status:SUCCESS 2023-11-27 13:38:12.254 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000020, status:SUCCESS 2023-11-27 13:38:12.885 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:38:12.885279+08:00, lastProcessTime=2023-11-27T13:38:02.831310+08:00 2023-11-27 13:38:13.007 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:38:13.008 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:38:13.053 INFO 10 --- [pool-ocp-schedules-19,,9f098e7349a7] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-11-27 13:38:13.053 INFO 10 --- [pool-Check host status8,,a5dedec3b522] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=133.197.206.97, originalStatus=OFFLINE 2023-11-27 13:38:13.055 INFO 10 --- [pool-Check host status1,,c98cc9b879e7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=133.197.204.3, originalStatus=OFFLINE 2023-11-27 13:38:13.055 INFO 10 --- [pool-Check host status6,,14c2c2772cea] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=133.197.206.92, originalStatus=OFFLINE 2023-11-27 13:38:13.055 INFO 10 --- [pool-Check host status4,,5213b03b59f2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=133.197.204.10, originalStatus=ONLINE 2023-11-27 13:38:13.055 INFO 10 --- [pool-Check host status5,,9ed5bf540210] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=133.197.204.11, originalStatus=ONLINE 2023-11-27 13:38:13.055 INFO 10 --- [pool-Check host status7,,9ae4e6f57b78] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=6, ip=133.197.204.12, originalStatus=ONLINE 2023-11-27 13:38:13.055 INFO 10 --- [pool-Check host status13,,8cc85eb39495] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=133.197.204.13, originalStatus=ONLINE 2023-11-27 13:38:13.055 INFO 10 --- [pool-Check host status12,,c2607e834cfa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=9, ip=133.197.206.88, originalStatus=OFFLINE 2023-11-27 13:38:13.062 INFO 10 --- [pool-metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1701063420, nodeCount=1000, affectRows=1000 2023-11-27 13:38:13.066 INFO 10 --- [pool-metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1701063420, nodeCount=1000, validCount=1000 2023-11-27 13:38:13.079 INFO 10 --- [pool-Check host status13,,8cc85eb39495] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2023-11-27 13:38:13.080 INFO 10 --- [pool-Check host status5,,9ed5bf540210] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-11-27 13:38:13.081 INFO 10 --- [pool-Check host status6,,14c2c2772cea] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:38:13.082 INFO 10 --- [pool-Check host status7,,9ae4e6f57b78] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2023-11-27 13:38:13.086 INFO 10 --- [pool-Check host status8,,a5dedec3b522] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:38:13.086 INFO 10 --- [pool-Check host status4,,5213b03b59f2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-11-27 13:38:13.087 INFO 10 --- [pool-Check host status12,,c2607e834cfa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:38:13.128 INFO 10 --- [pool-Check host status1,,c98cc9b879e7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:38:13.324 INFO 10 --- [pool-metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1701063420, nodeCount=707, affectRows=707 2023-11-27 13:38:13.326 INFO 10 --- [pool-metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1701063420, nodeCount=707, validCount=707 2023-11-27 13:38:13.337 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:38:12.885279+08:00 2023-11-27 13:38:13.446 INFO 10 --- [pool-Check host status12,,c2607e834cfa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.88:62888/api/v1/time, params:null 2023-11-27 13:38:13.450 WARN 10 --- [pool-Check host status12,,c2607e834cfa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check failed. host=9, ip=133.197.206.88, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.88:62888 [/133.197.206.88] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.getHostTime(AgentExecutor.java:127) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.doCheckHostStatus(HostCheckManagerImpl.java:206) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostStatus(HostCheckManagerImpl.java:168) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.88:62888 [/133.197.206.88] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 24 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 33 common frames omitted 2023-11-27 13:38:13.454 INFO 10 --- [pool-Check host status6,,14c2c2772cea] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.92:62888/api/v1/time, params:null 2023-11-27 13:38:13.455 INFO 10 --- [pool-Check host status12,,c2607e834cfa] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [9:133.197.206.88], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:38:13.455 INFO 10 --- [pool-Check host status12,,c2607e834cfa] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:38:13.456 INFO 10 --- [pool-Check host status13,,8cc85eb39495] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.13:62888/api/v1/time, params:null 2023-11-27 13:38:13.456 INFO 10 --- [pool-Check host status12,,c2607e834cfa] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=9 2023-11-27 13:38:13.456 INFO 10 --- [pool-Check host status12,,c2607e834cfa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=9 2023-11-27 13:38:13.457 WARN 10 --- [pool-Check host status6,,14c2c2772cea] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check failed. host=3, ip=133.197.206.92, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.92:62888 [/133.197.206.92] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.getHostTime(AgentExecutor.java:127) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.doCheckHostStatus(HostCheckManagerImpl.java:206) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostStatus(HostCheckManagerImpl.java:168) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.92:62888 [/133.197.206.92] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 24 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 33 common frames omitted 2023-11-27 13:38:13.459 INFO 10 --- [pool-Check host status12,,c2607e834cfa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:38:13.462 INFO 10 --- [pool-Check host status8,,a5dedec3b522] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.97:62888/api/v1/time, params:null 2023-11-27 13:38:13.462 INFO 10 --- [pool-Check host status7,,9ae4e6f57b78] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.12:62888/api/v1/time, params:null 2023-11-27 13:38:13.464 WARN 10 --- [pool-Check host status8,,a5dedec3b522] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check failed. host=1, ip=133.197.206.97, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.97:62888 [/133.197.206.97] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.getHostTime(AgentExecutor.java:127) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.doCheckHostStatus(HostCheckManagerImpl.java:206) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostStatus(HostCheckManagerImpl.java:168) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.97:62888 [/133.197.206.97] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 24 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 33 common frames omitted 2023-11-27 13:38:13.465 INFO 10 --- [pool-Check host status13,,8cc85eb39495] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=7, ip=133.197.204.13, timestamp=2023-11-27T00:38:13.444825182-05:00 2023-11-27 13:38:13.466 INFO 10 --- [pool-Check host status7,,9ae4e6f57b78] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=6, ip=133.197.204.12, timestamp=2023-11-27T00:38:13.461544534-05:00 2023-11-27 13:38:13.470 INFO 10 --- [pool-Check host status7,,9ae4e6f57b78] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=6, lastAvailableTime=2023-11-27T13:38:13.466+08:00 2023-11-27 13:38:13.471 INFO 10 --- [pool-Check host status6,,14c2c2772cea] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [3:133.197.206.92], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:38:13.471 INFO 10 --- [pool-Check host status6,,14c2c2772cea] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:38:13.476 INFO 10 --- [pool-Check host status7,,9ae4e6f57b78] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 13 last available time 2023-11-27T13:38:13.466+08:00 and status AVAILABLE 2023-11-27 13:38:13.482 INFO 10 --- [pool-Check host status13,,8cc85eb39495] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2023-11-27T13:38:13.465+08:00 2023-11-27 13:38:13.489 INFO 10 --- [pool-Check host status8,,a5dedec3b522] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [1:133.197.206.97], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:38:13.489 INFO 10 --- [pool-Check host status8,,a5dedec3b522] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:38:13.489 INFO 10 --- [pool-Check host status5,,9ed5bf540210] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.11:62888/api/v1/time, params:null 2023-11-27 13:38:13.490 INFO 10 --- [pool-Check host status6,,14c2c2772cea] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=3 2023-11-27 13:38:13.490 INFO 10 --- [pool-Check host status6,,14c2c2772cea] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2023-11-27 13:38:13.495 INFO 10 --- [pool-Check host status5,,9ed5bf540210] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=133.197.204.11, timestamp=2023-11-27T00:38:13.491078181-05:00 2023-11-27 13:38:13.495 INFO 10 --- [pool-Check host status5,,9ed5bf540210] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-11-27T13:38:13.495+08:00 2023-11-27 13:38:13.498 INFO 10 --- [pool-Check host status13,,8cc85eb39495] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 14 last available time 2023-11-27T13:38:13.465+08:00 and status AVAILABLE 2023-11-27 13:38:13.499 INFO 10 --- [pool-Check host status5,,9ed5bf540210] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 12 last available time 2023-11-27T13:38:13.495+08:00 and status AVAILABLE 2023-11-27 13:38:13.500 INFO 10 --- [pool-Check host status1,,c98cc9b879e7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.3:62888/api/v1/time, params:null 2023-11-27 13:38:13.511 WARN 10 --- [pool-Check host status1,,c98cc9b879e7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check failed. host=2, ip=133.197.204.3, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.204.3:62888 [/133.197.204.3] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.getHostTime(AgentExecutor.java:127) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.doCheckHostStatus(HostCheckManagerImpl.java:206) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostStatus(HostCheckManagerImpl.java:168) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.204.3:62888 [/133.197.204.3] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 24 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 33 common frames omitted 2023-11-27 13:38:13.515 INFO 10 --- [pool-Check host status6,,14c2c2772cea] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:38:13.516 INFO 10 --- [pool-Check host status1,,c98cc9b879e7] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [2:133.197.204.3], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:38:13.516 INFO 10 --- [pool-Check host status1,,c98cc9b879e7] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:38:13.519 INFO 10 --- [pool-Check host status8,,a5dedec3b522] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=1 2023-11-27 13:38:13.519 INFO 10 --- [pool-Check host status8,,a5dedec3b522] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2023-11-27 13:38:13.524 INFO 10 --- [pool-Check host status1,,c98cc9b879e7] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=2 2023-11-27 13:38:13.524 INFO 10 --- [pool-Check host status1,,c98cc9b879e7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2023-11-27 13:38:13.527 INFO 10 --- [pool-Check host status13,,8cc85eb39495] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2023-11-27T13:38:13.524+08:00 2023-11-27 13:38:13.528 INFO 10 --- [pool-Check host status1,,c98cc9b879e7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:38:13.530 INFO 10 --- [pool-Check host status13,,8cc85eb39495] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 14 last available time 2023-11-27T13:38:13.524+08:00 and status AVAILABLE 2023-11-27 13:38:13.535 INFO 10 --- [pool-Check host status8,,a5dedec3b522] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:38:13.536 INFO 10 --- [pool-Check host status13,,8cc85eb39495] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=7 2023-11-27 13:38:13.543 INFO 10 --- [pool-Check host status4,,5213b03b59f2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.10:62888/api/v1/time, params:null 2023-11-27 13:38:13.546 INFO 10 --- [pool-Check host status7,,9ae4e6f57b78] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=6, lastAvailableTime=2023-11-27T13:38:13.542+08:00 2023-11-27 13:38:13.547 INFO 10 --- [pool-Check host status4,,5213b03b59f2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=133.197.204.10, timestamp=2023-11-27T00:38:13.540088816-05:00 2023-11-27 13:38:13.549 INFO 10 --- [pool-Check host status4,,5213b03b59f2] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-11-27T13:38:13.547+08:00 2023-11-27 13:38:13.567 INFO 10 --- [pool-Check host status7,,9ae4e6f57b78] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 13 last available time 2023-11-27T13:38:13.542+08:00 and status AVAILABLE 2023-11-27 13:38:13.568 INFO 10 --- [pool-Check host status13,,8cc85eb39495] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2023-11-27 13:38:13.569 INFO 10 --- [pool-Check host status4,,5213b03b59f2] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 11 last available time 2023-11-27T13:38:13.547+08:00 and status AVAILABLE 2023-11-27 13:38:13.595 INFO 10 --- [pool-Check host status5,,9ed5bf540210] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-11-27T13:38:13.594+08:00 2023-11-27 13:38:13.597 INFO 10 --- [pool-Check host status7,,9ae4e6f57b78] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=6 2023-11-27 13:38:13.598 INFO 10 --- [pool-Check host status5,,9ed5bf540210] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 12 last available time 2023-11-27T13:38:13.594+08:00 and status AVAILABLE 2023-11-27 13:38:13.602 INFO 10 --- [pool-Check host status4,,5213b03b59f2] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-11-27T13:38:13.602+08:00 2023-11-27 13:38:13.607 INFO 10 --- [pool-Check host status4,,5213b03b59f2] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 11 last available time 2023-11-27T13:38:13.602+08:00 and status AVAILABLE 2023-11-27 13:38:13.612 INFO 10 --- [pool-Check host status5,,9ed5bf540210] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-11-27 13:38:13.624 INFO 10 --- [pool-Check host status4,,5213b03b59f2] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-11-27 13:38:13.634 INFO 10 --- [pool-Check host status7,,9ae4e6f57b78] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2023-11-27 13:38:13.634 INFO 10 --- [pool-Check host status5,,9ed5bf540210] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-11-27 13:38:13.640 INFO 10 --- [pool-Check host status4,,5213b03b59f2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-11-27 13:38:13.653 INFO 10 --- [pool-Check host status13,,8cc85eb39495] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-11-27 13:38:13.687 INFO 10 --- [pool-Check host status4,,5213b03b59f2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-11-27 13:38:13.691 INFO 10 --- [pool-Check host status5,,9ed5bf540210] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-11-27 13:38:13.695 INFO 10 --- [pool-Check host status7,,9ae4e6f57b78] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-11-27 13:38:13.700 INFO 10 --- [pool-Check host status13,,8cc85eb39495] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2023-11-27 13:38:13.715 INFO 10 --- [pool-Check host status13,,8cc85eb39495] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2023-11-27 13:38:13.732 INFO 10 --- [pool-Check host status4,,5213b03b59f2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=31 2023-11-27 13:38:13.739 INFO 10 --- [pool-Check host status5,,9ed5bf540210] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=34 2023-11-27 13:38:13.741 INFO 10 --- [pool-Check host status7,,9ae4e6f57b78] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=37 2023-11-27 13:38:13.782 INFO 10 --- [pool-Check host status7,,9ae4e6f57b78] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=37 2023-11-27 13:38:13.783 INFO 10 --- [pool-Check host status4,,5213b03b59f2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=31 2023-11-27 13:38:13.790 INFO 10 --- [pool-Check host status5,,9ed5bf540210] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=34 2023-11-27 13:38:13.792 INFO 10 --- [pool-Check host status13,,8cc85eb39495] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-11-27 13:38:13.824 INFO 10 --- [pool-Check host status5,,9ed5bf540210] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-11-27 13:38:13.828 INFO 10 --- [pool-Check host status4,,5213b03b59f2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-11-27 13:38:13.830 INFO 10 --- [pool-Check host status7,,9ae4e6f57b78] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-11-27 13:38:13.835 INFO 10 --- [pool-Check host status13,,8cc85eb39495] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2023-11-27 13:38:13.846 INFO 10 --- [pool-Check host status13,,8cc85eb39495] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2023-11-27 13:38:13.864 INFO 10 --- [pool-Check host status5,,9ed5bf540210] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=35 2023-11-27 13:38:13.871 INFO 10 --- [pool-Check host status4,,5213b03b59f2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=32 2023-11-27 13:38:13.880 INFO 10 --- [pool-Check host status13,,8cc85eb39495] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-11-27 13:38:13.883 INFO 10 --- [pool-Check host status7,,9ae4e6f57b78] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=38 2023-11-27 13:38:13.883 INFO 10 --- [pool-Check host status5,,9ed5bf540210] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=35 2023-11-27 13:38:13.893 INFO 10 --- [pool-Check host status4,,5213b03b59f2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=32 2023-11-27 13:38:13.897 INFO 10 --- [pool-Check host status7,,9ae4e6f57b78] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=38 2023-11-27 13:38:13.915 INFO 10 --- [pool-Check host status5,,9ed5bf540210] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-11-27 13:38:13.916 INFO 10 --- [pool-Check host status4,,5213b03b59f2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-11-27 13:38:13.928 INFO 10 --- [pool-Check host status7,,9ae4e6f57b78] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-11-27 13:38:13.934 INFO 10 --- [pool-Check host status13,,8cc85eb39495] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2023-11-27 13:38:13.946 INFO 10 --- [pool-Check host status13,,8cc85eb39495] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2023-11-27 13:38:13.955 INFO 10 --- [pool-Check host status5,,9ed5bf540210] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=36 2023-11-27 13:38:13.956 INFO 10 --- [pool-Check host status4,,5213b03b59f2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=33 2023-11-27 13:38:13.973 INFO 10 --- [pool-Check host status7,,9ae4e6f57b78] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=39 2023-11-27 13:38:13.980 INFO 10 --- [pool-Check host status4,,5213b03b59f2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=33 2023-11-27 13:38:13.987 INFO 10 --- [pool-Check host status5,,9ed5bf540210] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=36 2023-11-27 13:38:13.999 INFO 10 --- [pool-Check host status7,,9ae4e6f57b78] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=39 2023-11-27 13:38:14.033 INFO 10 --- [pool-Check host status15,,b738e1a72d21] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=133.197.204.3, originalStatus=OFFLINE 2023-11-27 13:38:14.033 INFO 10 --- [pool-Check host status14,,3362442b70ff] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=133.197.206.97, originalStatus=OFFLINE 2023-11-27 13:38:14.033 INFO 10 --- [pool-Check host status11,,8cb2b243ba26] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=133.197.206.92, originalStatus=OFFLINE 2023-11-27 13:38:14.034 INFO 10 --- [pool-Check host status9,,079fe6613d1c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=133.197.204.10, originalStatus=ONLINE 2023-11-27 13:38:14.035 INFO 10 --- [pool-Check host status10,,2f173c2a89a1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=133.197.204.11, originalStatus=ONLINE 2023-11-27 13:38:14.035 INFO 10 --- [pool-Check host status16,,e14918cf915d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=6, ip=133.197.204.12, originalStatus=ONLINE 2023-11-27 13:38:14.035 INFO 10 --- [pool-Check host status17,,35b2aa7248e1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=133.197.204.13, originalStatus=ONLINE 2023-11-27 13:38:14.035 INFO 10 --- [pool-Check host status19,,4052b3e0cca5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=9, ip=133.197.206.88, originalStatus=OFFLINE 2023-11-27 13:38:14.041 INFO 10 --- [pool-Check host status15,,b738e1a72d21] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:38:14.041 INFO 10 --- [pool-Check host status14,,3362442b70ff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:38:14.042 INFO 10 --- [pool-Check host status17,,35b2aa7248e1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2023-11-27 13:38:14.043 INFO 10 --- [pool-Check host status11,,8cb2b243ba26] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:38:14.067 INFO 10 --- [pool-Check host status9,,079fe6613d1c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-11-27 13:38:14.069 INFO 10 --- [pool-Check host status10,,2f173c2a89a1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-11-27 13:38:14.073 INFO 10 --- [pool-Check host status19,,4052b3e0cca5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:38:14.074 INFO 10 --- [pool-Check host status16,,e14918cf915d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2023-11-27 13:38:14.097 INFO 10 --- [pool-Check host status15,,b738e1a72d21] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.3:62888/api/v1/agent/status, params:null 2023-11-27 13:38:14.101 WARN 10 --- [pool-Check host status15,,b738e1a72d21] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent status check failed. host=2, ip=133.197.204.3, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.204.3:62888 [/133.197.204.3] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.agentStatus(AgentExecutor.java:622) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostAgentStatus(HostCheckManagerImpl.java:83) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.204.3:62888 [/133.197.204.3] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 23 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 32 common frames omitted 2023-11-27 13:38:14.122 INFO 10 --- [pool-Check host status15,,b738e1a72d21] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [2:133.197.204.3], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:38:14.122 INFO 10 --- [pool-Check host status15,,b738e1a72d21] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:38:14.133 INFO 10 --- [pool-Check host status15,,b738e1a72d21] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=2 2023-11-27 13:38:14.133 INFO 10 --- [pool-Check host status15,,b738e1a72d21] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2023-11-27 13:38:14.134 INFO 10 --- [pool-Check host status10,,2f173c2a89a1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.11:62888/api/v1/agent/status, params:null 2023-11-27 13:38:14.140 INFO 10 --- [pool-Check host status10,,2f173c2a89a1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=133.197.204.11 2023-11-27 13:38:14.140 INFO 10 --- [pool-Check host status10,,2f173c2a89a1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=133.197.204.11 2023-11-27 13:38:14.140 INFO 10 --- [pool-Check host status10,,2f173c2a89a1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=133.197.204.11 2023-11-27 13:38:14.141 INFO 10 --- [pool-Check host status15,,b738e1a72d21] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:38:14.144 INFO 10 --- [pool-Check host status11,,8cb2b243ba26] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.92:62888/api/v1/agent/status, params:null 2023-11-27 13:38:14.148 WARN 10 --- [pool-Check host status11,,8cb2b243ba26] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent status check failed. host=3, ip=133.197.206.92, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.92:62888 [/133.197.206.92] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.agentStatus(AgentExecutor.java:622) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostAgentStatus(HostCheckManagerImpl.java:83) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.92:62888 [/133.197.206.92] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 23 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 32 common frames omitted 2023-11-27 13:38:14.177 INFO 10 --- [pool-ocp-schedules-13,,1f5165f834f9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:38:14.178 INFO 10 --- [pool-Check host status17,,35b2aa7248e1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.13:62888/api/v1/agent/status, params:null 2023-11-27 13:38:14.184 INFO 10 --- [pool-Check host status17,,35b2aa7248e1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=133.197.204.13 2023-11-27 13:38:14.185 INFO 10 --- [pool-Check host status17,,35b2aa7248e1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=133.197.204.13 2023-11-27 13:38:14.185 INFO 10 --- [pool-Check host status17,,35b2aa7248e1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=133.197.204.13 2023-11-27 13:38:14.194 INFO 10 --- [pool-Check host status11,,8cb2b243ba26] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [3:133.197.206.92], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:38:14.194 INFO 10 --- [pool-Check host status11,,8cb2b243ba26] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:38:14.201 INFO 10 --- [pool-Check host status11,,8cb2b243ba26] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=3 2023-11-27 13:38:14.201 INFO 10 --- [pool-Check host status11,,8cb2b243ba26] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2023-11-27 13:38:14.204 INFO 10 --- [pool-Check host status11,,8cb2b243ba26] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:38:14.207 INFO 10 --- [pool-Check host status19,,4052b3e0cca5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.88:62888/api/v1/agent/status, params:null 2023-11-27 13:38:14.213 WARN 10 --- [pool-Check host status19,,4052b3e0cca5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent status check failed. host=9, ip=133.197.206.88, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.88:62888 [/133.197.206.88] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.agentStatus(AgentExecutor.java:622) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostAgentStatus(HostCheckManagerImpl.java:83) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.88:62888 [/133.197.206.88] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 23 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 32 common frames omitted 2023-11-27 13:38:14.223 INFO 10 --- [pool-Check host status19,,4052b3e0cca5] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [9:133.197.206.88], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:38:14.223 INFO 10 --- [pool-Check host status19,,4052b3e0cca5] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:38:14.231 INFO 10 --- [pool-Check host status19,,4052b3e0cca5] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=9 2023-11-27 13:38:14.231 INFO 10 --- [pool-Check host status19,,4052b3e0cca5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=9 2023-11-27 13:38:14.235 INFO 10 --- [pool-Check host status19,,4052b3e0cca5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:38:14.235 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:38:14.263 INFO 10 --- [pool-Check host status9,,079fe6613d1c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.10:62888/api/v1/agent/status, params:null 2023-11-27 13:38:14.268 INFO 10 --- [pool-Check host status9,,079fe6613d1c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=133.197.204.10 2023-11-27 13:38:14.268 INFO 10 --- [pool-Check host status9,,079fe6613d1c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=133.197.204.10 2023-11-27 13:38:14.268 INFO 10 --- [pool-Check host status9,,079fe6613d1c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=133.197.204.10 2023-11-27 13:38:14.271 INFO 10 --- [pool-Check host status16,,e14918cf915d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.12:62888/api/v1/agent/status, params:null 2023-11-27 13:38:14.275 INFO 10 --- [pool-Check host status16,,e14918cf915d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=133.197.204.12 2023-11-27 13:38:14.275 INFO 10 --- [pool-Check host status16,,e14918cf915d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=133.197.204.12 2023-11-27 13:38:14.275 INFO 10 --- [pool-Check host status16,,e14918cf915d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=133.197.204.12 2023-11-27 13:38:14.276 INFO 10 --- [pool-Check host status14,,3362442b70ff] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.97:62888/api/v1/agent/status, params:null 2023-11-27 13:38:14.278 WARN 10 --- [pool-Check host status14,,3362442b70ff] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent status check failed. host=1, ip=133.197.206.97, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.97:62888 [/133.197.206.97] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.agentStatus(AgentExecutor.java:622) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostAgentStatus(HostCheckManagerImpl.java:83) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.97:62888 [/133.197.206.97] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 23 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 32 common frames omitted 2023-11-27 13:38:14.284 INFO 10 --- [pool-Check host status14,,3362442b70ff] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [1:133.197.206.97], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:38:14.285 INFO 10 --- [pool-Check host status14,,3362442b70ff] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:38:14.289 INFO 10 --- [pool-Check host status14,,3362442b70ff] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=1 2023-11-27 13:38:14.289 INFO 10 --- [pool-Check host status14,,3362442b70ff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2023-11-27 13:38:14.292 INFO 10 --- [pool-Check host status14,,3362442b70ff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:38:14.955 INFO 10 --- [alarm-detect-15,,ec389d69056e] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_cluster_exists_inactive_server detect time range 1701063474 ~ 1701063484 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={inactive_server_count > 0=[ExpressionEvaluateResult(matchValue=0.0, metric=inactive_server_count, match=true, value=1.0, values={1701063480=1.0}, valueCount=1, labels={app=OB, ob_cluster_id=10, ob_cluster_name=darenOB, server_ips=133.197.204.3})]}, matched=true, message=null, level=2) 2023-11-27 13:38:15.051 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:38:15.081 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:38:15.106 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063495, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:15.106 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063495, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:15.107 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063495, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:15.107 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063495, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:15.116 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063495, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:15.161 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:38:15.681 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:38:15.770 INFO 10 --- [pool-ocp-schedules-17,,de062139262d] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:38:15.770 INFO 10 --- [pool-ocp-schedules-17,,de062139262d] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:38:16.009 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:38:16.016 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:38:16.262 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:38:17.667 INFO 10 --- [http-nio-0.0.0.0-8080-exec-7,f7262b7592b640b0,a7cfce851740] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.11, traceId=f7262b7592b640b0, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:38:17.759 INFO 10 --- [http-nio-0.0.0.0-8080-exec-7,f7262b7592b640b0,a7cfce851740] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.11, traceId=f7262b7592b640b0, duration=92 ms] 2023-11-27 13:38:18.352 INFO 10 --- [http-nio-0.0.0.0-8080-exec-6,2db8740751e74b9b,0c46b1fc4f91] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.12, traceId=2db8740751e74b9b, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:38:18.360 INFO 10 --- [http-nio-0.0.0.0-8080-exec-6,2db8740751e74b9b,0c46b1fc4f91] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.12, traceId=2db8740751e74b9b, duration=9 ms] 2023-11-27 13:38:19.002 INFO 10 --- [http-nio-0.0.0.0-8080-exec-1,6bea144b456c496a,0228e7c0bd1f] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.12, traceId=6bea144b456c496a, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:38:19.019 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:38:19.026 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:38:19.027 INFO 10 --- [http-nio-0.0.0.0-8080-exec-1,6bea144b456c496a,0228e7c0bd1f] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.12, traceId=6bea144b456c496a, duration=26 ms] 2023-11-27 13:38:19.071 INFO 10 --- [pool-prometheus-exporter-check2,,3bebcd7aeece] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/stat, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:38:19.072 WARN 10 --- [pool-prometheus-exporter-check2,,3bebcd7aeece] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/stat is inactive 2023-11-27 13:38:19.074 INFO 10 --- [pool-prometheus-exporter-check2,,3bebcd7aeece] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62888/metrics/stat, message=java.net.ConnectException: Connection refused: /133.197.204.3:62888 2023-11-27 13:38:19.075 WARN 10 --- [pool-prometheus-exporter-check2,,3bebcd7aeece] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62888/metrics/stat is inactive 2023-11-27 13:38:19.076 INFO 10 --- [pool-prometheus-exporter-check2,,3bebcd7aeece] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/node/host, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:38:19.076 WARN 10 --- [pool-prometheus-exporter-check2,,3bebcd7aeece] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/node/host is inactive 2023-11-27 13:38:19.079 INFO 10 --- [pool-prometheus-exporter-check2,,3bebcd7aeece] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/obproxy, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:38:19.079 WARN 10 --- [pool-prometheus-exporter-check2,,3bebcd7aeece] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/obproxy is inactive 2023-11-27 13:38:19.082 INFO 10 --- [pool-prometheus-exporter-check2,,3bebcd7aeece] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/node/obproxy, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:38:19.083 WARN 10 --- [pool-prometheus-exporter-check2,,3bebcd7aeece] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/node/obproxy is inactive 2023-11-27 13:38:19.085 INFO 10 --- [pool-prometheus-exporter-check2,,3bebcd7aeece] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/ob/extra, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:38:19.086 WARN 10 --- [pool-prometheus-exporter-check2,,3bebcd7aeece] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/ob/extra is inactive 2023-11-27 13:38:19.086 INFO 10 --- [pool-prometheus-exporter-check2,,3bebcd7aeece] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/node/ob, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:38:19.087 WARN 10 --- [pool-prometheus-exporter-check2,,3bebcd7aeece] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/node/ob is inactive 2023-11-27 13:38:19.087 INFO 10 --- [pool-prometheus-exporter-check2,,3bebcd7aeece] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/ob/basic, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:38:19.088 WARN 10 --- [pool-prometheus-exporter-check2,,3bebcd7aeece] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/ob/basic is inactive 2023-11-27 13:38:20.103 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063500, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:20.105 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063500, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:20.106 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063500, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:20.106 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063500, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:20.107 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063500, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:20.270 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:37:20.0, nextRun=2023-11-27 13:38:20.0, currentDate=Mon Nov 27 13:38:20 CST 2023 2023-11-27 13:38:20.270 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-11-27 13:37:20.0, nextRunTime=2023-11-27 13:38:20.0} 2023-11-27 13:38:20.318 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-11-27 13:38:20.318 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:38:20.318 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-11-27 13:38:20.409 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000056, operation:EXECUTE, state: RUNNING 2023-11-27 13:38:20.412 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000056, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-11-27T13:38:20.407+08:00, endTime=null} 2023-11-27 13:38:20.448 INFO 10 --- [http-nio-0.0.0.0-8080-exec-3,8c64fe150e054dd4,3a4ba08dab18] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.13, traceId=8c64fe150e054dd4, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:38:20.454 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,41277eaf6560] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000056, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000047, task_operation=execute, latest_execution_start_time=2023-11-27T13:38:20.407+08:00, sub_task_instance_id=10000056}, listMap={}}, executor=10.120.0.1 2023-11-27 13:38:20.474 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,41277eaf6560] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-11-27 13:38:20.474 INFO 10 --- [ForkJoinPool-29-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-11-27 13:38:20.494 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,41277eaf6560] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000056, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:38:20.495 INFO 10 --- [http-nio-0.0.0.0-8080-exec-3,8c64fe150e054dd4,3a4ba08dab18] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.13, traceId=8c64fe150e054dd4, duration=47 ms] 2023-11-27 13:38:20.607 INFO 10 --- [ForkJoinPool-29-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:20.608 INFO 10 --- [ForkJoinPool-29-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:20.608 INFO 10 --- [ForkJoinPool-29-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:20.608 INFO 10 --- [ForkJoinPool-29-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:20.609 INFO 10 --- [ForkJoinPool-29-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:20.609 INFO 10 --- [ForkJoinPool-29-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:20.610 INFO 10 --- [ForkJoinPool-29-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT MAX(CASE `name` WHEN 'cluster' THEN `info` ELSE '' END ) `cluster`, MAX(CASE `name` WHEN 'merge_status' THEN `info` ELSE '' END ) `merge_status`, MAX(CASE `name` WHEN 'global_broadcast_version' THEN `value` ELSE 0 END ) `global_broadcast_version`, MAX(CASE `name` WHEN 'frozen_version' THEN `value` ELSE 0 END ) `frozen_version`, MAX(CASE `name` WHEN 'last_merged_version' THEN `value` ELSE 0 END ) `last_merged_version`, MAX(CASE `name` WHEN 'frozen_time' THEN `value` ELSE 0 END ) `frozen_time`, MAX(CASE `name` WHEN 'is_merge_error' THEN `value` ELSE 0 END ) `merge_error` FROM oceanbase.__all_zone WHERE `zone` = '' 2023-11-27 13:38:20.612 INFO 10 --- [ForkJoinPool-29-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:20.612 INFO 10 --- [ForkJoinPool-29-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:20.613 INFO 10 --- [ForkJoinPool-29-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select column_value from oceanbase.__all_core_table where table_name ='__all_global_stat' and column_name = 'frozen_version' 2023-11-27 13:38:20.614 INFO 10 --- [ForkJoinPool-29-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:20.614 INFO 10 --- [ForkJoinPool-29-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:20.615 INFO 10 --- [ForkJoinPool-29-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, MAX(CASE `name` WHEN 'region' THEN `info` ELSE '' END ) `region`, MAX(CASE `name` WHEN 'idc' THEN `info` ELSE '' END ) `idc`, MAX(CASE `name` WHEN 'status' THEN `info` ELSE '' END ) `status`, MAX(CASE `name` WHEN 'merge_status' THEN `info` ELSE '' END ) `merge_status`, MAX(CASE `name` WHEN 'broadcast_version' THEN `value` ELSE 0 END ) `broadcast_version`, MAX(CASE `name` WHEN 'all_merged_version' THEN `value` ELSE 0 END ) `all_merged_version`, MAX(CASE `name` WHEN 'last_merged_version' THEN `value` ELSE 0 END ) `last_merged_version`, MAX(CASE `name` WHEN 'merge_start_time' THEN `value` ELSE 0 END ) `merge_start_time`, MAX(CASE `name` WHEN 'last_merged_time' THEN `value` ELSE 0 END ) `last_merged_time`, MAX(CASE `name` WHEN 'is_merge_timeout' THEN `value` ELSE 0 END ) `merge_timeout` FROM oceanbase.__all_zone WHERE `zone` <> '' GROUP BY `zone` 2023-11-27 13:38:20.616 INFO 10 --- [ForkJoinPool-29-worker-2,,] s.t.b.c.CollectAllClustersCompactionTask : compaction collect success, cluster=ObClusterOverview(id=1, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"133.197.206.97:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1700793672036000}, createTime=2022-11-29T14:52:53+08:00, creator=admin, obVersion=3.1.4, tenantCount=3) 2023-11-27 13:38:20.753 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000047, status:SUCCESS 2023-11-27 13:38:20.754 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000047, status:SUCCESS 2023-11-27 13:38:21.793 INFO 10 --- [pool-overview-container-7,,f949f9a4aaf8] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:38:21.891 INFO 10 --- [pool-scanner--11448-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=100 2023-11-27 13:38:22.026 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:38:22.027 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:38:22.774 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:38:22.773564+08:00, lastProcessTime=2023-11-27T13:38:12.885279+08:00 2023-11-27 13:38:22.874 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:38:22.773564+08:00 2023-11-27 13:38:23.494 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.120.0.1:8080 2023-11-27 13:38:23.666 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.120.0.1:8080 2023-11-27 13:38:23.898 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.120.0.1:8080 2023-11-27 13:38:24.235 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:38:24.264 INFO 10 --- [pool-ocp-schedules-19,,93b32c0160ee] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:38:24.549 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.120.0.1:8080 2023-11-27 13:38:24.602 INFO 10 --- [alarm-detect-3,,79809049f383] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_cluster_exists_inactive_server detect time range 1701063484 ~ 1701063494 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={inactive_server_count > 0=[ExpressionEvaluateResult(matchValue=0.0, metric=inactive_server_count, match=true, value=1.0, values={1701063490=1.0}, valueCount=1, labels={app=OB, ob_cluster_id=10, ob_cluster_name=darenOB, server_ips=133.197.204.3})]}, matched=true, message=null, level=2) 2023-11-27 13:38:25.027 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:38:25.029 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:38:25.070 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:38:25.083 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:38:25.102 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063505, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:25.104 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063505, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:25.104 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063505, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:25.106 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063505, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:25.106 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063505, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:25.178 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:38:25.703 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:38:25.735 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:37:26.0, nextRun=2023-11-27 13:38:25.0, currentDate=Mon Nov 27 13:38:25 CST 2023 2023-11-27 13:38:25.736 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-11-27 13:37:26.0, nextRunTime=2023-11-27 13:38:25.0} 2023-11-27 13:38:25.819 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-11-27 13:38:25.819 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:38:25.819 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-11-27 13:38:26.266 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:38:26.524 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000057, operation:EXECUTE, state: RUNNING 2023-11-27 13:38:26.527 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000057, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-11-27T13:38:26.521+08:00, endTime=null} 2023-11-27 13:38:26.613 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,c6769fdd57a9] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000057, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000048, task_operation=execute, latest_execution_start_time=2023-11-27T13:38:26.521+08:00, sub_task_instance_id=10000057}, listMap={host_ids=[1, 2, 3, 4, 5, 6, 7, 9]}}, executor=10.120.0.1 2023-11-27 13:38:26.729 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,c6769fdd57a9] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270931, maxInactiveSeconds=180 2023-11-27 13:38:26.729 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,c6769fdd57a9] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270931, maxInactiveSeconds=180 2023-11-27 13:38:26.729 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,c6769fdd57a9] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270931, maxInactiveSeconds=180 2023-11-27 13:38:26.729 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,c6769fdd57a9] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270931, maxInactiveSeconds=180 2023-11-27 13:38:26.729 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,c6769fdd57a9] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270931, maxInactiveSeconds=180 2023-11-27 13:38:26.729 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,c6769fdd57a9] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270931, maxInactiveSeconds=180 2023-11-27 13:38:26.729 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,c6769fdd57a9] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270931, maxInactiveSeconds=180 2023-11-27 13:38:26.729 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,c6769fdd57a9] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270931, maxInactiveSeconds=180 2023-11-27 13:38:26.729 WARN 10 --- [pool-subtask-executor15,554481c3fd1b426b,c6769fdd57a9] c.o.o.s.t.b.host.CheckExporterTask : Host [id=1, ip=133.197.206.97, status=OFFLINE] not online, skip check exporter 2023-11-27 13:38:26.729 WARN 10 --- [pool-subtask-executor15,554481c3fd1b426b,c6769fdd57a9] c.o.o.s.t.b.host.CheckExporterTask : Host [id=2, ip=133.197.204.3, status=OFFLINE] not online, skip check exporter 2023-11-27 13:38:26.729 WARN 10 --- [pool-subtask-executor15,554481c3fd1b426b,c6769fdd57a9] c.o.o.s.t.b.host.CheckExporterTask : Host [id=3, ip=133.197.206.92, status=OFFLINE] not online, skip check exporter 2023-11-27 13:38:26.732 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,c6769fdd57a9] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=133.197.204.10] desired exporter count: 8, actual exporter count: 8 2023-11-27 13:38:26.733 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,c6769fdd57a9] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=133.197.204.11] desired exporter count: 8, actual exporter count: 8 2023-11-27 13:38:26.734 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,c6769fdd57a9] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=6, ip=133.197.204.12] desired exporter count: 8, actual exporter count: 8 2023-11-27 13:38:26.737 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,c6769fdd57a9] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=7, ip=133.197.204.13] desired exporter count: 8, actual exporter count: 8 2023-11-27 13:38:26.737 WARN 10 --- [pool-subtask-executor15,554481c3fd1b426b,c6769fdd57a9] c.o.o.s.t.b.host.CheckExporterTask : Host [id=9, ip=133.197.206.88, status=OFFLINE] not online, skip check exporter 2023-11-27 13:38:26.744 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,c6769fdd57a9] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000057, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:38:26.944 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000048, status:SUCCESS 2023-11-27 13:38:26.945 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000048, status:SUCCESS 2023-11-27 13:38:27.018 INFO 10 --- [http-nio-0.0.0.0-8080-exec-5,109231d87d0f48f3,a1c0cb14dd4a] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.10, traceId=109231d87d0f48f3, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:38:27.063 INFO 10 --- [http-nio-0.0.0.0-8080-exec-5,109231d87d0f48f3,a1c0cb14dd4a] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.10, traceId=109231d87d0f48f3, duration=46 ms] 2023-11-27 13:38:27.500 INFO 10 --- [http-nio-0.0.0.0-8080-exec-7,06ee6fc2c9bc4e80,7d98653199cb] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.13, traceId=06ee6fc2c9bc4e80, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:38:27.516 INFO 10 --- [http-nio-0.0.0.0-8080-exec-7,06ee6fc2c9bc4e80,7d98653199cb] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.13, traceId=06ee6fc2c9bc4e80, duration=17 ms] 2023-11-27 13:38:28.029 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:38:28.031 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:38:29.216 INFO 10 --- [http-nio-0.0.0.0-8080-exec-6,ca75eedf75ea48f4,b9e89b2b1a55] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.10, traceId=ca75eedf75ea48f4, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:38:29.244 INFO 10 --- [http-nio-0.0.0.0-8080-exec-6,ca75eedf75ea48f4,b9e89b2b1a55] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.10, traceId=ca75eedf75ea48f4, duration=29 ms] 2023-11-27 13:38:29.482 INFO 10 --- [http-nio-0.0.0.0-8080-exec-1,859358a00622401f,1219b0262605] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=133.197.206.97, traceId=859358a00622401f, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-11-27 13:38:29.484 INFO 10 --- [http-nio-0.0.0.0-8080-exec-1,859358a00622401f,1219b0262605] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.206.97, traceId=859358a00622401f, duration=3 ms] 2023-11-27 13:38:30.027 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:37:30.0, nextRun=2023-11-27 13:38:30.0, currentDate=Mon Nov 27 13:38:30 CST 2023 2023-11-27 13:38:30.027 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-11-27 13:37:30.0, nextRunTime=2023-11-27 13:38:30.0} 2023-11-27 13:38:30.037 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-11-27 13:38:30.037 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:38:30.037 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:38:30.037 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-11-27 13:38:30.052 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:38:00.0, nextRun=2023-11-27 13:38:30.0, currentDate=Mon Nov 27 13:38:30 CST 2023 2023-11-27 13:38:30.053 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-11-27 13:38:00.0, nextRunTime=2023-11-27 13:38:30.0} 2023-11-27 13:38:30.063 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-11-27 13:38:30.063 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:38:30.063 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-11-27 13:38:30.106 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063510, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:30.106 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063510, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:30.106 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063510, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:30.106 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063510, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:30.106 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063510, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:30.584 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000058, operation:EXECUTE, state: RUNNING 2023-11-27 13:38:30.587 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000058, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-11-27T13:38:30.582+08:00, endTime=null} 2023-11-27 13:38:30.610 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000059, operation:EXECUTE, state: RUNNING 2023-11-27 13:38:30.613 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000059, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-11-27T13:38:30.606+08:00, endTime=null} 2023-11-27 13:38:30.629 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000060, operation:EXECUTE, state: RUNNING 2023-11-27 13:38:30.638 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000058, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=8000049, task_operation=execute, latest_execution_start_time=2023-11-27T13:38:30.582+08:00, sub_task_instance_id=10000058}, listMap={cluster_ids=[1, 2]}}, executor=10.120.0.1 2023-11-27 13:38:30.640 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000060, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-11-27T13:38:30.625+08:00, endTime=null} 2023-11-27 13:38:30.659 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000059, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=8000049, task_operation=execute, latest_execution_start_time=2023-11-27T13:38:30.606+08:00, sub_task_instance_id=10000059}, listMap={cluster_ids=[1, 2]}}, executor=10.120.0.1 2023-11-27 13:38:30.744 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c9d7cd2d69d7] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000060, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000050, task_operation=execute, latest_execution_start_time=2023-11-27T13:38:30.625+08:00, sub_task_instance_id=10000060}, listMap={}}, executor=10.120.0.1 2023-11-27 13:38:30.744 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c9d7cd2d69d7] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-11-27 13:38:30.799 INFO 10 --- [pool-763-thread-1,554481c3fd1b426b,e64953cf125a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:darenOB, tenant:sys 2023-11-27 13:38:30.801 INFO 10 --- [pool-763-thread-2,554481c3fd1b426b,99ab4ac0c5b1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:darenOB, tenant:daren 2023-11-27 13:38:30.823 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:30.827 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:30.827 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:30.830 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:30.831 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:30.831 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:30.833 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, tenant_name, (CASE WHEN tenant_id = 1 THEN 'SYS' ELSE 'USER' END) AS tenant_type, (CASE WHEN compatibility_mode = 1 THEN 'ORACLE' ELSE 'MYSQL' END) AS compatibility_mode, zone_list, primary_zone, locality, status, locked, read_only, in_recyclebin, TIMESTAMPDIFF(SECOND, gmt_create, now()) AS exist_seconds FROM oceanbase.__all_tenant 2023-11-27 13:38:30.840 INFO 10 --- [pool-763-thread-1,554481c3fd1b426b,e64953cf125a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:30.841 INFO 10 --- [pool-763-thread-1,554481c3fd1b426b,e64953cf125a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:30.841 INFO 10 --- [pool-763-thread-1,554481c3fd1b426b,e64953cf125a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:30.844 INFO 10 --- [pool-763-thread-1,554481c3fd1b426b,e64953cf125a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:30.849 INFO 10 --- [pool-763-thread-1,554481c3fd1b426b,e64953cf125a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:30.849 INFO 10 --- [pool-763-thread-1,554481c3fd1b426b,e64953cf125a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:30.850 INFO 10 --- [pool-763-thread-1,554481c3fd1b426b,e64953cf125a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-11-27 13:38:30.851 INFO 10 --- [pool-763-thread-2,554481c3fd1b426b,99ab4ac0c5b1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:30.851 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:30.851 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:30.851 INFO 10 --- [pool-763-thread-2,554481c3fd1b426b,99ab4ac0c5b1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:30.851 INFO 10 --- [pool-763-thread-2,554481c3fd1b426b,99ab4ac0c5b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:30.851 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1] 2023-11-27 13:38:30.854 INFO 10 --- [pool-763-thread-2,554481c3fd1b426b,99ab4ac0c5b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:30.863 INFO 10 --- [pool-763-thread-2,554481c3fd1b426b,99ab4ac0c5b1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:30.863 INFO 10 --- [pool-763-thread-2,554481c3fd1b426b,99ab4ac0c5b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:30.864 INFO 10 --- [pool-763-thread-2,554481c3fd1b426b,99ab4ac0c5b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-11-27 13:38:30.864 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:30.864 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:30.865 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1] 2023-11-27 13:38:30.875 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:30.875 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:30.875 INFO 10 --- [pool-763-thread-1,554481c3fd1b426b,e64953cf125a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1701021601748952808, broadcast scn:1701021601748952808, saved frozen scn:1701021601748952808 2023-11-27 13:38:30.876 INFO 10 --- [pool-763-thread-1,554481c3fd1b426b,e64953cf125a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:darenOB, tenant:sys 2023-11-27 13:38:30.876 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1001] 2023-11-27 13:38:30.907 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:30.910 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:30.910 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:30.911 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:30.914 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:30.914 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:30.915 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-11-27 13:38:30.919 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:30.919 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:30.920 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-11-27 13:38:30.925 INFO 10 --- [pool-763-thread-2,554481c3fd1b426b,99ab4ac0c5b1] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1701021600396785720, broadcast scn:1701021600396785720, saved frozen scn:1701021600396785720 2023-11-27 13:38:30.925 INFO 10 --- [pool-763-thread-2,554481c3fd1b426b,99ab4ac0c5b1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:darenOB, tenant:daren 2023-11-27 13:38:30.936 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:30.936 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:30.936 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c9d7cd2d69d7] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-11-27 13:38:30.938 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-11-27 13:38:30.961 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:30.961 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:30.967 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:30.968 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:30.968 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-11-27 13:38:30.981 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1001] 2023-11-27 13:38:30.995 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:30.995 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:30.999 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c9d7cd2d69d7] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000060, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:38:31.002 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-11-27 13:38:31.012 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:31.012 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:31.018 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-11-27 13:38:31.023 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1002] 2023-11-27 13:38:31.038 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:38:31.042 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:31.042 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:31.042 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:38:31.045 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1002] 2023-11-27 13:38:31.073 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-11-27 13:38:31.141 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000050, status:SUCCESS 2023-11-27 13:38:31.147 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000050, status:SUCCESS 2023-11-27 13:38:31.172 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:38:31.173 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:31.174 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:38:31.174 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:31.174 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:31.175 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:31.177 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:31.177 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:31.177 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-11-27 13:38:31.180 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:31.184 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:31.184 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:31.188 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:31.200 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:31.200 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:31.205 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-11-27 13:38:31.215 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:38:31.216 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:31.216 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:31.216 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:31.217 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:31.219 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:31.219 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:31.219 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:38:31.279 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:38:31.280 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:31.284 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:31.284 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:31.293 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:31.296 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:31.296 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:31.298 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:38:31.302 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1001, ALTER_TENANT_LOCALITY] 2023-11-27 13:38:31.307 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:31.311 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:31.311 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:31.316 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:31.320 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:38:31.320 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:31.320 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:31.321 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:31.321 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:38:31.321 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:31.321 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:31.322 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:31.323 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:31.323 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:31.324 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1001, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:38:31.373 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:38:31.377 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:31.380 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:31.380 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:31.382 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:31.387 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:31.387 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:31.388 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-11-27 13:38:31.403 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:38:31.405 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:31.408 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:31.408 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:31.409 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:31.419 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:31.419 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:31.420 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-11-27 13:38:31.453 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:38:31.455 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:38:31.456 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:31.456 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:31.456 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:31.459 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:31.459 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:31.460 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:31.460 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:31.463 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:31.463 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:31.463 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:31.464 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:38:31.465 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-11-27 13:38:31.469 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:31.469 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:31.471 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:38:31.472 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-11-27 13:38:31.563 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.120.0.1 2023-11-27 13:38:31.568 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.120.0.1 2023-11-27 13:38:31.637 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:38:31.638 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:31.639 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:31.639 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:31.639 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:31.644 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:31.644 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:31.644 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1] 2023-11-27 13:38:31.654 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:38:31.680 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-11-27 13:38:31.680 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:31.680 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:31.685 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1001] 2023-11-27 13:38:31.702 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:38:31.706 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-11-27 13:38:31.707 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:31.707 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:31.708 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1002] 2023-11-27 13:38:31.720 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:38:31.738 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:38:31.743 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:31.745 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:31.745 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:31.747 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-11-27 13:38:31.747 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-11-27 13:38:31.747 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=obcluster, obClusterId=1 2023-11-27 13:38:31.753 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:31.759 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:31.760 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:31.761 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-11-27 13:38:31.761 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,cd8aad199734] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000058, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:38:31.771 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:38:31.776 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-11-27 13:38:31.777 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:31.777 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:31.779 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-11-27 13:38:31.791 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:38:31.795 INFO 10 --- [pool-overview-container-1,,e5e159d1ddf3] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:38:31.810 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-11-27 13:38:31.810 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-11-27 13:38:31.810 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=darenOB, obClusterId=10 2023-11-27 13:38:31.815 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,e76e1a479f0f] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000059, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:38:31.899 INFO 10 --- [pool-scanner--11541-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=100 2023-11-27 13:38:32.270 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000049, status:SUCCESS 2023-11-27 13:38:32.271 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000049, status:SUCCESS 2023-11-27 13:38:32.752 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:38:32.751955+08:00, lastProcessTime=2023-11-27T13:38:22.773564+08:00 2023-11-27 13:38:32.877 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:38:32.751955+08:00 2023-11-27 13:38:33.974 INFO 10 --- [pool-prometheus-cache-monitor1,,7d8a3127e081] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":3414,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":1707,"seriesIdKeySize":87133,"secondWriteQueue":0,"minuteWriteCacheSize":32687} 2023-11-27 13:38:33.974 INFO 10 --- [pool-prometheus-cache-monitor1,,7d8a3127e081] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_zone_current_timestamp=38, ob_server_resource_memory_assigned_bytes=21, tcp_retrans=30, ob_unit_num=21, go_memstats_mspan_sys_bytes=17, go_memstats_stack_sys_bytes=17, exporter_export_metrics_bytes_total=28, node_filesystem_size_bytes=87, http_server_requests_seconds_count=40, ob_compaction_frozen_time=18, ob_system_event_total_waits=34, monagent_input_sql_duration_seconds_sum=60, observer_fd_count=20, ob_plan_cache_hit_total=28, processor_process_metrics_total=22, node_filesystem_readonly=86, ob_server_resource_disk_in_use_bytes=19, go_memstats_frees_total=17, sql_audit_input_collect_data_jumped_total=12, io_await=60, host_available=13, go_memstats_mallocs_total=17, mysql_output_write_sql_failed_count=23, ob_unit_config_max_memory_bytes=45, io_util=60, ob_system_event_detail_total_timeouts=782, ob_tenant_disk_v4_data_size=18, ob_latch_sleep_total=326, ob_tenant_unit_min_memory=41, ob_memstore_total_bytes=28, process_resident_memory_bytes=17, ob_cache_size_bytes=47, ob_tenant_unit_max_memory=41, ob_tenant_assigned_cpu_total=25, input_collect_metrics_total=29, ob_tenant_server_available_disk_free=28, ob_tenant_server_avg_cpu_used_percentage=23, node_disk_io_time_weighted_seconds_total=39, node_disk_write_time_seconds_total=39, cpu_count=30, output_write_metrics_total=15, ob_cluster_sync=175, process_open_fds=17, odp_transaction_total=22, ob_system_event_detail_time_waited_us=782, process_max_fds=17, ob_server_resource_cpu_assigned_percent=20, ocp_alarm_detect_duration_ms_count=82, go_memstats_heap_inuse_bytes=17, ob_tenant_server_max_cpu_used_percentage=23, proxy_memory_used_bytes=30, go_memstats_mcache_sys_bytes=17, ob_tenant_disk_v4_log_size=18, node_memory_memfree_bytes=29, process_exists=38, go_gc_duration_seconds=22, ob_server_resource_disk_bytes=21, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=27, node_memory_cached_bytes=29, ob_table_num=24, ob_disk_free_bytes=21, node_filesystem_avail_bytes=87, ob_backup_storage_capacity_bytes=349, ob_active_session_num=27, ob_unit_config_max_iops=45, ob_plan_cache_memory_bytes=28, node_network_transmit_errs_total=65, go_memstats_last_gc_time_seconds=17, go_gc_duration_seconds_count=17, monitor_exporter_avaliable=78, ob_tenant_server_max_memory_used_percentage=23, node_disk_read_bytes_total=39, mysql_output_write_sql_milliseconds_summary_count=25, ob_compaction_is_suspended=18, ob_tenant_unit_log_disk_size=22, go_memstats_lookups_total=17, mysql_output_write_sql_count=25, log_tailer_count=17, host_agent_process_status=21, ob_server_resource_cpu=21, go_memstats_alloc_bytes=17, ob_server_num=58, node_cpu_seconds_total_native=47, ob_tenant_unit_count=39, ob_session_all_num=25, ob_tenant_resource_min_memory=25, ob_memstore_freeze_times=28, ob_system_event_time_waited=34, ob_tenant_resource_min_cpu=25, ob_cluster_status_check=49, monagent_plugin_execute_seconds_total=41, node_net_bandwidth_bps=45, monagent_oceanbase_db_pool_idle=7, go_memstats_heap_idle_bytes=17, go_memstats_gc_sys_bytes=17, ob_server_resource_memory_bytes=21, ob_tenant_sys_partition_required_size=12, ob_latch_get_total=326, go_info=18, ob_tenant_assigned_mem_assigned=25, log_tailer_processed_file_count=24, node_load15=29, ob_tenant_server_required_size=35, ob_unit_config_min_iops=45, ob_server_partition_replica_limit=7, ob_tenant_unit_max_cpu=41, ob_latch_wait_seconds_total=326, node_memory_buffers_bytes=29, go_memstats_heap_released_bytes=17, node_network_receive_bytes_total=65, ob_memstore_freeze_trigger_bytes=28, ob_waitevent_wait_seconds_total=28, ob_server_stopped_duration_seconds=20, exposer_request_latencies_count=8, ob_role_is_rootserver=20, go_memstats_next_gc_bytes=17, mysql_output_context_timeout_discard_metrics_count=17, ob_session_active_num=25, node_filesystem_files_free=109, ob_memtable_max_snapshot_duration_seconds=27, node_disk_read_time_seconds_total=39, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=22, process_virtual_memory_bytes=17, mysql_output_write_sql_retry_count=23, ob_tenant_assigned_cpu_assigned=25, ob_tenant_unit_min_cpu=41, ob_unit_config_max_cpu=45, ob_unit_config_max_session_num=18, http_request_milliseconds_summary_count=79, monagent_oceanbase_audit_timeinfo_cache_size=7, net_connectivity=10, monagent_input_sql_duration_seconds_bucket=70, ob_latch_immediate_get_total=326, node_cpu_seconds_total=37, http_request_milliseconds_summary=82, ob_latch_immediate_miss_total=326, ob_index_error_num=16, ob_tenant_server_available_disk_total=28, ob_latch_spin_get_total=326, ob_tenant_memory_hold_bytes=18, go_memstats_buck_hash_sys_bytes=17, monagent_pipeline_report_metrics_total=18, mysql_output_write_sql_milliseconds_summary_sum=25, ob_tenant_disk_total_size=25, exposer_transferred_bytes_total=8, odp_sql_request_total=67, mysql_output_write_task_size=24, ob_tenant_context_memory_count=339, node_disk_reads_completed_total=39, exporter_export_metrics_total=28, ob_sysstat=129, ob_partition_num=13, node_ntp_offset_seconds=29, ob_latch_miss_total=326, go_memstats_heap_sys_bytes=17, ocp_meta_db_health=4, ob_disk_total_bytes=21, ob_zone_stat=38, go_memstats_heap_objects=17, go_memstats_mcache_inuse_bytes=17, go_memstats_other_sys_bytes=17, go_threads=17, http_request_milliseconds_summary_sum=79, ob_memstore_active_bytes=28, http_server_requests_seconds_sum=40, go_memstats_alloc_bytes_total=17, ob_tenant_server_partition_replica_count=35, ob_tenant_assigned_mem_total=25, node_memory_memtotal_bytes=29, node_network_transmit_drop_total=65, ob_system_event_detail_total_waits=782, odp_used_connections=28, ob_tenant_task_max_duration_seconds=25, backup_storage_capacity_used_percent=14, exposer_request_latencies=11, ob_waitevent_wait_total=28, odp_sql_cost_total=66, ob_tenant500_memory_used_bytes=20, node_network_transmit_bytes_total=65, ob_compaction_is_error=18, node_network_receive_drop_total=65, ob_server_partition_replica_count=7, monagent_pipeline_execute_total=27, node_filesystem_files=109, ob_tenant_server_avg_memory_used_percentage=23, process_start_time_seconds=17, go_memstats_heap_alloc_bytes=17, ob_tenant_resource_max_cpu=25, go_gc_duration_seconds_sum=17, go_memstats_mspan_inuse_bytes=17, mysql_output_write_sql_milliseconds_summary=28, ob_server_resource_memory_assigned_percent=20, monagent_pipeline_execute_seconds_total=27, go_memstats_sys_bytes=17, node_network_receive_errs_total=65, ob_all_session_num=27, observer_ulimit_max_fd_count=20, ob_plan_cache_access_total=28, ob_tenant_server_row_count=35, ob_tenant_memory_used_bytes=18, monagent_oceanbase_plan_cache_size=7, ob_tenant_server_data_size=35, go_memstats_gc_cpu_fraction=17, monagent_oceanbase_audit_cache_size=7, ob_tenant_context_memory_used_bytes=339, node_disk_writes_completed_total=39, node_load1=29, process_cpu_seconds_total=17, ob_tenant_resource_max_memory=25, go_memstats_stack_inuse_bytes=17, ob_tenant_user_partition_required_size=12, node_load5=29, monagent_input_sql_duration_seconds_count=60, node_disk_written_bytes_total=39, monagent_oceanbase_ash_cache_size=7, ob_tenant_context_memory_hold_bytes=339, go_goroutines=17, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=21, exposer_request_latencies_sum=8, ob_unit_config_min_cpu=45, process_virtual_memory_max_bytes=17, monagent_oceanbase_plan_timeinfo_cache_size=7, monagent_plugin_execute_total=41, ob_tenant500_memory_hold_bytes=20, exposer_scrapes_total=8, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=22, ob_compaction_start_time=18, oceanbase_connectivity=21} 2023-11-27 13:38:34.043 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:38:34.048 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:38:34.236 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:38:34.321 INFO 10 --- [pool-ocp-schedules-12,,a1592acae863] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:38:34.504 INFO 10 --- [alarm-detect-6,,c12a7642fb53] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_cluster_exists_inactive_server detect time range 1701063494 ~ 1701063504 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={inactive_server_count > 0=[ExpressionEvaluateResult(matchValue=0.0, metric=inactive_server_count, match=true, value=1.0, values={1701063500=1.0}, valueCount=1, labels={app=OB, ob_cluster_id=10, ob_cluster_name=darenOB, server_ips=133.197.204.3})]}, matched=true, message=null, level=2) 2023-11-27 13:38:35.072 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:38:35.095 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:38:35.102 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063515, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:35.103 INFO 10 --- [pool-prometheus-second-schedule-1,,6a1dc7f53cbf] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=14, collectTaskCount=14 2023-11-27 13:38:35.104 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063515, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:35.104 INFO 10 --- [pool-prometheus-second-schedule-1,,c1431dedbda3] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=11, collectTaskCount=11 2023-11-27 13:38:35.106 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063515, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:35.106 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063515, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:35.109 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063515, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:35.196 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:38:35.710 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:38:35.801 INFO 10 --- [pool-ocp-schedules-11,,e4d821d0de48] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:38:35.801 INFO 10 --- [pool-ocp-schedules-11,,e4d821d0de48] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:38:36.272 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:38:37.048 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:38:37.058 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:38:39.239 INFO 10 --- [http-nio-0.0.0.0-8080-exec-3,895429f446aa4ade,fda29650d908] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.11, traceId=895429f446aa4ade, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:38:39.306 INFO 10 --- [http-nio-0.0.0.0-8080-exec-3,895429f446aa4ade,fda29650d908] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.11, traceId=895429f446aa4ade, duration=68 ms] 2023-11-27 13:38:39.584 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.120.0.1:8080 2023-11-27 13:38:40.058 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:38:40.063 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:38:40.102 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063520, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:40.105 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063520, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:40.108 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063520, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:40.108 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063520, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:40.111 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063520, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:41.800 INFO 10 --- [pool-overview-container-6,,bb3bffae50c7] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:38:41.978 INFO 10 --- [pool-scanner--11634-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=100 2023-11-27 13:38:42.011 INFO 10 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2023-11-27T13:34:40.064, end=2023-11-27T13:36:42.006} 2023-11-27 13:38:42.082 INFO 10 --- [pool-13-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 6 diagnose targets Pair(left=Tenant(id=5, name=daren, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1,zone3,zone4,zone5,zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:34:40.064, end=2023-11-27T13:36:42.006},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-11-27 13:38:42.194 INFO 10 --- [pool-13-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=daren, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1,zone3,zone4,zone5,zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null) 2023-11-27 13:38:42.212 INFO 10 --- [pool-13-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 9 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1;zone3;zone4;zone5;zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:34:40.064, end=2023-11-27T13:36:42.006},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-11-27 13:38:42.244 INFO 10 --- [pool-13-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1;zone3;zone4;zone5;zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null) 2023-11-27 13:38:42.247 INFO 10 --- [pool-13-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1001, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:34:40.064, end=2023-11-27T13:36:42.006},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-11-27 13:38:42.247 INFO 10 --- [pool-13-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=meta_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1001, sqlAuditEnabled=null), right=null) 2023-11-27 13:38:42.249 INFO 10 --- [pool-13-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:34:40.064, end=2023-11-27T13:36:42.006},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-11-27 13:38:42.249 INFO 10 --- [pool-13-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null) 2023-11-27 13:38:42.251 INFO 10 --- [pool-13-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:34:40.064, end=2023-11-27T13:36:42.006},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-11-27 13:38:42.251 INFO 10 --- [pool-13-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1, sqlAuditEnabled=null), right=null) 2023-11-27 13:38:42.251 INFO 10 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2023-11-27 13:38:42.760 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:38:42.759958+08:00, lastProcessTime=2023-11-27T13:38:32.751955+08:00 2023-11-27 13:38:42.866 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:38:42.759958+08:00 2023-11-27 13:38:43.064 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:38:43.065 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:38:44.235 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:38:44.360 INFO 10 --- [pool-ocp-schedules-19,,7da365053b6b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:38:44.537 INFO 10 --- [alarm-detect-14,,36058a2d59fa] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_cluster_exists_inactive_server detect time range 1701063504 ~ 1701063514 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={inactive_server_count > 0=[ExpressionEvaluateResult(matchValue=0.0, metric=inactive_server_count, match=true, value=1.0, values={1701063510=1.0}, valueCount=1, labels={app=OB, ob_cluster_id=10, ob_cluster_name=darenOB, server_ips=133.197.204.3})]}, matched=true, message=null, level=2) 2023-11-27 13:38:45.095 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:38:45.102 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063525, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:45.102 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063525, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:45.104 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063525, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:45.113 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063525, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:45.119 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063525, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:45.132 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:38:45.220 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:38:45.716 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:38:46.065 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:38:46.066 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:38:46.276 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:38:49.066 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:38:49.068 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:38:50.103 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063530, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:50.104 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063530, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:50.104 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063530, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:50.104 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063530, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:50.104 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063530, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:50.433 INFO 10 --- [http-nio-0.0.0.0-8080-exec-5,8bf1d2283d3f4ce3,74155d9e7b4c] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.11, traceId=8bf1d2283d3f4ce3, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:38:50.471 INFO 10 --- [http-nio-0.0.0.0-8080-exec-5,8bf1d2283d3f4ce3,74155d9e7b4c] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.11, traceId=8bf1d2283d3f4ce3, duration=38 ms] 2023-11-27 13:38:51.572 INFO 10 --- [pool-ocp-schedules-18,,b90abb4ef730] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-11-27 13:38:51.588 INFO 10 --- [pool-ocp-schedules-18,,b90abb4ef730] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-11-27 13:38:51.797 INFO 10 --- [pool-overview-container-2,,226b44f9e35b] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:38:51.870 INFO 10 --- [pool-scanner--11727-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=100 2023-11-27 13:38:52.068 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:38:52.069 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:38:52.226 INFO 10 --- [pool-ocp-schedules-12,,26e0cd8919d1] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-11-27 13:38:52.228 INFO 10 --- [pool-ocp-schedules-12,,26e0cd8919d1] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-11-27 13:38:52.230 INFO 10 --- [pool-ocp-schedules-18,,47be72d37a05] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-11-27 13:38:52.231 INFO 10 --- [pool-ocp-schedules-18,,47be72d37a05] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-11-27 13:38:52.233 INFO 10 --- [pool-ocp-schedules-4,,ce39f2adb85b] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-11-27 13:38:52.237 INFO 10 --- [pool-ocp-schedules-4,,ce39f2adb85b] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-11-27 13:38:52.293 INFO 10 --- [ForkJoinPool-32-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:52.294 INFO 10 --- [ForkJoinPool-32-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:52.294 INFO 10 --- [ForkJoinPool-32-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:52.295 INFO 10 --- [ForkJoinPool-32-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:52.301 INFO 10 --- [ForkJoinPool-32-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:52.301 INFO 10 --- [ForkJoinPool-32-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:52.303 INFO 10 --- [ForkJoinPool-32-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:52.306 INFO 10 --- [ForkJoinPool-32-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-11-27 13:38:52.312 INFO 10 --- [ForkJoinPool-32-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:52.312 INFO 10 --- [ForkJoinPool-32-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:52.312 INFO 10 --- [ForkJoinPool-32-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:52.313 INFO 10 --- [ForkJoinPool-32-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:52.314 INFO 10 --- [ForkJoinPool-32-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:52.314 INFO 10 --- [ForkJoinPool-32-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:52.314 INFO 10 --- [ForkJoinPool-32-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:52.315 INFO 10 --- [ForkJoinPool-32-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=3.1.4 2023-11-27 13:38:52.321 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:52.322 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:52.322 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:52.322 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:52.323 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:52.324 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:52.324 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, MAX(CASE `name` WHEN 'region' THEN `info` ELSE '' END ) `region`, MAX(CASE `name` WHEN 'idc' THEN `info` ELSE '' END ) `idc`, MAX(CASE `name` WHEN 'status' THEN `info` ELSE '' END ) `status`, MAX(CASE `name` WHEN 'merge_status' THEN `info` ELSE '' END ) `merge_status`, MAX(CASE `name` WHEN 'broadcast_version' THEN `value` ELSE 0 END ) `broadcast_version`, MAX(CASE `name` WHEN 'all_merged_version' THEN `value` ELSE 0 END ) `all_merged_version`, MAX(CASE `name` WHEN 'last_merged_version' THEN `value` ELSE 0 END ) `last_merged_version`, MAX(CASE `name` WHEN 'merge_start_time' THEN `value` ELSE 0 END ) `merge_start_time`, MAX(CASE `name` WHEN 'last_merged_time' THEN `value` ELSE 0 END ) `last_merged_time`, MAX(CASE `name` WHEN 'is_merge_timeout' THEN `value` ELSE 0 END ) `merge_timeout` FROM oceanbase.__all_zone WHERE `zone` <> '' GROUP BY `zone` 2023-11-27 13:38:52.361 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:52.362 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:52.362 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:52.362 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:52.366 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:52.366 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:52.371 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-11-27 13:38:52.594 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:38:52.594 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:52.595 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:52.595 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:52.595 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:52.597 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:52.597 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:52.598 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, MAX(CASE `name` WHEN 'region' THEN `info` ELSE '' END ) `region`, MAX(CASE `name` WHEN 'idc' THEN `info` ELSE '' END ) `idc`, MAX(CASE `name` WHEN 'status' THEN `info` ELSE '' END ) `status`, MAX(CASE `name` WHEN 'merge_status' THEN `info` ELSE '' END ) `merge_status`, MAX(CASE `name` WHEN 'broadcast_version' THEN `value` ELSE 0 END ) `broadcast_version`, MAX(CASE `name` WHEN 'all_merged_version' THEN `value` ELSE 0 END ) `all_merged_version`, MAX(CASE `name` WHEN 'last_merged_version' THEN `value` ELSE 0 END ) `last_merged_version`, MAX(CASE `name` WHEN 'merge_start_time' THEN `value` ELSE 0 END ) `merge_start_time`, MAX(CASE `name` WHEN 'last_merged_time' THEN `value` ELSE 0 END ) `last_merged_time`, MAX(CASE `name` WHEN 'is_merge_timeout' THEN `value` ELSE 0 END ) `merge_timeout` FROM oceanbase.__all_zone WHERE `zone` <> '' GROUP BY `zone` 2023-11-27 13:38:52.599 WARN 10 --- [ForkJoinPool-31-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-11-27 13:38:52.600 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:52.600 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:52.601 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, stop_time, start_service_time, block_migrate_in_time, id FROM oceanbase.__all_server ORDER BY zone ASC, id ASC 2023-11-27 13:38:52.602 INFO 10 --- [ForkJoinPool-31-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.206.97:2882, status=ACTIVE, startServiceTime=1701063018577428, active=true 2023-11-27 13:38:52.602 INFO 10 --- [ForkJoinPool-31-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=133.197.206.97:2882, stopTime=0, stopped=false 2023-11-27 13:38:52.602 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=IDLE, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=IDLE, startTime=null, stopTime=null} 2023-11-27 13:38:52.621 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=IDLE, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:38:52.622 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=133.197.206.97, port=2882, sqlPort=2881, version=3.1.4_10000092022071511-b4bfa011ceaef428782dcb65ae89190c40b78c2f(Jul 15 2022 11:45:14), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-27T13:30:19+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.206.97, port=2882, sqlPort=2881, version=3.1.4_10000092022071511-b4bfa011ceaef428782dcb65ae89190c40b78c2f(Jul 15 2022 11:45:14), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-27T05:30:18.577428Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:38:52.640 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=133.197.206.97, port=2882, sqlPort=2881, version=3.1.4_10000092022071511-b4bfa011ceaef428782dcb65ae89190c40b78c2f(Jul 15 2022 11:45:14), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-27T05:30:18.577428Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:38:52.699 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:38:52.705 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:52.706 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:52.706 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:52.706 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:52.708 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:52.708 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:52.708 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:52.755 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:38:52.756 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:52.756 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:52.756 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:52.757 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:52.760 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:52.761 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:52.762 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:52.766 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:52.766 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:52.767 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT cluster_id, cluster_name, created, cluster_role, cluster_status, current_scn, standby_became_primary_scn, primary_cluster_id, (time_to_usec(now(6)) - current_scn)/1000000 as delay_time, protection_mode, redo_transport_options, protection_level FROM oceanbase.v$ob_cluster 2023-11-27 13:38:52.768 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:38:52.768 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:52.768 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT switchover_status, switchover_info FROM oceanbase.v$ob_cluster 2023-11-27 13:38:52.782 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=obcluster, obClusterId=1 2023-11-27 13:38:52.782 INFO 10 --- [ForkJoinPool-31-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"133.197.206.97:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1700793672036000}, createTime=2022-11-29T14:52:53+08:00, creator=admin, obVersion=3.1.4, tenantCount=3) 2023-11-27 13:38:52.812 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:38:52.810744+08:00, lastProcessTime=2023-11-27T13:38:42.759958+08:00 2023-11-27 13:38:52.983 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:38:52.810744+08:00 2023-11-27 13:38:53.119 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:38:53.120 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:53.120 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:53.120 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:53.121 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:53.125 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:53.125 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:53.125 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-11-27 13:38:53.128 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:53.128 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:53.129 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-11-27 13:38:53.131 INFO 10 --- [ForkJoinPool-31-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.204.3:2882, status=INACTIVE, startServiceTime=0, active=false 2023-11-27 13:38:53.132 INFO 10 --- [ForkJoinPool-31-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.204.10:2882, status=ACTIVE, startServiceTime=1695730092226557, active=true 2023-11-27 13:38:53.132 INFO 10 --- [ForkJoinPool-31-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=133.197.204.10:2882, stopTime=0, stopped=false 2023-11-27 13:38:53.132 INFO 10 --- [ForkJoinPool-31-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.204.11:2882, status=ACTIVE, startServiceTime=1695730305487530, active=true 2023-11-27 13:38:53.132 INFO 10 --- [ForkJoinPool-31-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=133.197.204.11:2882, stopTime=0, stopped=false 2023-11-27 13:38:53.132 INFO 10 --- [ForkJoinPool-31-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.204.12:2882, status=ACTIVE, startServiceTime=1695730736949667, active=true 2023-11-27 13:38:53.132 INFO 10 --- [ForkJoinPool-31-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=133.197.204.12:2882, stopTime=0, stopped=false 2023-11-27 13:38:53.132 INFO 10 --- [ForkJoinPool-31-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.204.13:2882, status=ACTIVE, startServiceTime=1695731005656813, active=true 2023-11-27 13:38:53.132 INFO 10 --- [ForkJoinPool-31-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=133.197.204.13:2882, stopTime=0, stopped=false 2023-11-27 13:38:53.132 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=14, name=zone6, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone6, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-27 13:38:53.150 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=14, name=zone6, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:38:53.150 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=14, ip=133.197.204.13, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=7, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T20:23:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.204.13, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=7, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:23:25.656813Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:38:53.169 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=14, ip=133.197.204.13, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=7, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:23:25.656813Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:38:53.169 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=13, name=zone5, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone5, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-27 13:38:53.188 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=13, name=zone5, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:38:53.188 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=13, ip=133.197.204.12, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=6, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T20:18:57+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.204.12, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=6, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:18:56.949667Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:38:53.211 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=13, ip=133.197.204.12, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=6, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:18:56.949667Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:38:53.211 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=12, name=zone4, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone4, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-27 13:38:53.236 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=12, name=zone4, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:38:53.236 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=12, ip=133.197.204.11, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T20:11:45+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.204.11, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:11:45.487530Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:38:53.260 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=12, ip=133.197.204.11, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:11:45.487530Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:38:53.260 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=11, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-27 13:38:53.278 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=11, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:38:53.278 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=11, ip=133.197.204.10, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=true, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T20:08:12+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.204.10, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=true, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:08:12.226557Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:38:53.313 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=11, ip=133.197.204.10, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=true, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:08:12.226557Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:38:53.313 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-27 13:38:53.330 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:38:53.331 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=133.197.204.3, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=2, status=UNAVAILABLE, innerStatus=INACTIVE, startTime=1970-01-01T08:00+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.204.3, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=2, status=null, innerStatus=INACTIVE, startTime=1970-01-01T00:00Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:38:53.349 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=133.197.204.3, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=2, status=UNAVAILABLE, innerStatus=INACTIVE, startTime=1970-01-01T00:00Z, stopTime=1970-01-01T00:00Z}, oldStatus=UNAVAILABLE, newStatus=UNAVAILABLE 2023-11-27 13:38:53.368 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:38:53.369 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:53.370 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:53.370 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:53.370 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:53.374 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:38:53.374 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:38:53.375 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:38:53.412 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=darenOB, obClusterId=10 2023-11-27 13:38:53.412 INFO 10 --- [ForkJoinPool-31-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=darenOB, obClusterId=10, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"darenOB","ObClusterId":10,"ObRegion":"darenOB","ObRegionId":10,"ReadonlyRsList":[],"RsList":[{"address":"133.197.204.10:2882","role":"LEADER","sql_port":2881},{"address":"133.197.204.11:2882","role":"FOLLOWER","sql_port":2881},{"address":"133.197.204.12:2882","role":"FOLLOWER","sql_port":2881},{"address":"133.197.204.13:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1701063480849000}, createTime=2022-11-29T15:05:37+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-11-27 13:38:53.515 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.120.0.1:8080 2023-11-27 13:38:53.672 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.120.0.1:8080 2023-11-27 13:38:53.903 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.120.0.1:8080 2023-11-27 13:38:54.235 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:38:54.419 INFO 10 --- [pool-ocp-schedules-1,,3652dbc23f23] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:38:54.532 INFO 10 --- [alarm-detect-1,,05da517e2355] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_cluster_exists_inactive_server detect time range 1701063514 ~ 1701063524 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={inactive_server_count > 0=[ExpressionEvaluateResult(matchValue=0.0, metric=inactive_server_count, match=true, value=1.0, values={1701063520=1.0}, valueCount=1, labels={app=OB, ob_cluster_id=10, ob_cluster_name=darenOB, server_ips=133.197.204.3})]}, matched=true, message=null, level=2) 2023-11-27 13:38:54.603 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.120.0.1:8080 2023-11-27 13:38:55.072 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:38:55.077 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:38:55.105 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063535, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:55.106 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063535, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:55.106 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063535, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:55.108 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063535, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:55.109 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063535, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:38:55.132 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:38:55.144 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:38:55.251 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:38:55.485 INFO 10 --- [pool-scanner--11820-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=100 2023-11-27 13:38:55.531 INFO 10 --- [pool-scanner--11821-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=100 2023-11-27 13:38:55.553 INFO 10 --- [pool-scanner--11822-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=100 2023-11-27 13:38:55.607 INFO 10 --- [pool-ocp-schedules-2,,5a484d11944e] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:38:55.644 INFO 10 --- [pool-ocp-schedules-2,,5a484d11944e] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:38:55.730 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:38:55.780 INFO 10 --- [pool-ocp-schedules-2,,5a484d11944e] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:38:55.823 INFO 10 --- [pool-ocp-schedules-2,,5a484d11944e] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:38:55.909 INFO 10 --- [pool-ocp-schedules-11,,25c3b63b1b35] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:38:55.910 INFO 10 --- [pool-ocp-schedules-11,,25c3b63b1b35] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:38:55.916 INFO 10 --- [pool-ocp-schedules-2,,5a484d11944e] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:38:55.958 INFO 10 --- [pool-ocp-schedules-2,,5a484d11944e] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:38:56.279 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:38:58.078 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:38:58.079 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:39:00.000 INFO 10 --- [pool-ocp-schedules-4,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-11-27 13:39:00.000 INFO 10 --- [pool-ocp-schedules-4,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-11-27 13:39:00.004 INFO 10 --- [pool-ocp-schedules-17,,] c.o.o.s.task.scheduled.OcpSchedules : Refresh OCP properties. 2023-11-27 13:39:00.004 INFO 10 --- [pool-ocp-schedules-11,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-11-27 13:39:00.005 INFO 10 --- [pool-ocp-schedules-17,,] c.o.ocp.core.property.PropertyService : Refresh config properties 2023-11-27 13:39:00.005 INFO 10 --- [pool-ocp-schedules-11,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1701063540 2023-11-27 13:39:00.010 INFO 10 --- [pool-ocp-schedules-17,,] c.o.o.b.spring.BootstrapRunListener : new: [] 2023-11-27 13:39:00.010 INFO 10 --- [pool-ocp-schedules-17,,] c.o.o.b.spring.BootstrapRunListener : init spring 2023-11-27 13:39:00.010 INFO 10 --- [pool-ocp-schedules-17,,] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2023-11-27 13:39:00.010 INFO 10 --- [pool-ocp-schedules-17,,] c.o.o.b.spring.BootstrapRunListener : starting 2023-11-27 13:39:00.015 INFO 10 --- [pool-ocp-schedules-17,,] c.o.o.b.spring.BootstrapRunListener : new: [] 2023-11-27 13:39:00.015 INFO 10 --- [pool-ocp-schedules-17,,] c.o.o.b.spring.BootstrapRunListener : init spring 2023-11-27 13:39:00.015 INFO 10 --- [pool-ocp-schedules-17,,] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2023-11-27 13:39:00.015 INFO 10 --- [pool-ocp-schedules-17,,] c.o.o.b.spring.BootstrapRunListener : starting 2023-11-27 13:39:00.048 INFO 10 --- [pool-ocp-schedules-4,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-11-27 13:39:00.104 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063540, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:00.105 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063540, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:00.105 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063540, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:00.105 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063540, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:00.105 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063540, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:00.943 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:38:00.0, nextRun=2023-11-27 13:39:00.0, currentDate=Mon Nov 27 13:39:00 CST 2023 2023-11-27 13:39:00.943 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-27 13:38:00.0, nextRunTime=2023-11-27 13:39:00.0} 2023-11-27 13:39:00.970 INFO 10 --- [pool-prometheus-minute-schedule-1,,3dc1fc549029] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1701063600, interval=60 2023-11-27 13:39:00.973 INFO 10 --- [pool-prometheus-minute-schedule-2,,9e0aeda932e1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1701063600, interval=60 2023-11-27 13:39:00.985 INFO 10 --- [pool-prometheus-minute-schedule-1,,3dc1fc549029] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=11, collectTaskCount=11 2023-11-27 13:39:00.985 INFO 10 --- [pool-prometheus-minute-schedule-1,,3dc1fc549029] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1701063540, count=11, elapsed=1699362477445 2023-11-27 13:39:00.986 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62888/metrics/stat, collectAt=1701063540, message=java.net.ConnectException: Connection refused: /133.197.206.88:62888, rootCause=ConnectException: Connection refused 2023-11-27 13:39:00.986 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62888/metrics/stat, collectAt=1701063540, message=java.net.ConnectException: Connection refused: /133.197.206.97:62888, rootCause=ConnectException: Connection refused 2023-11-27 13:39:00.987 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/extra, collectAt=1701063540, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:00.987 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/stat, collectAt=1701063540, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:00.989 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-11-27 13:39:00.989 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:39:00.989 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-11-27 13:39:01.001 INFO 10 --- [pool-prometheus-minute-schedule-2,,9e0aeda932e1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2023-11-27 13:39:01.002 INFO 10 --- [pool-prometheus-minute-schedule-2,,9e0aeda932e1] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1701063540, count=8, elapsed=1699362477462 2023-11-27 13:39:01.001 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/stat, collectAt=1701063540, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:01.002 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/stat, collectAt=1701063540, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:01.002 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:38:00.0, nextRun=2023-11-27 13:39:00.0, currentDate=Mon Nov 27 13:39:01 CST 2023 2023-11-27 13:39:01.003 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-27 13:38:00.0, nextRunTime=2023-11-27 13:39:00.0} 2023-11-27 13:39:01.008 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62888/metrics/stat, collectAt=1701063540, message=java.net.ConnectException: Connection refused: /133.197.206.92:62888, rootCause=ConnectException: Connection refused 2023-11-27 13:39:01.025 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-11-27 13:39:01.025 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:39:01.025 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-11-27 13:39:01.026 INFO 10 --- [pool-ocp-schedules-17,,] o.s.cloud.commons.util.InetUtils : Cannot determine local hostname 2023-11-27 13:39:01.079 INFO 10 --- [pool-ocp-schedules-17,,] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2023-11-27 13:39:01.087 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:39:01.092 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:39:01.110 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:38:00.0, nextRun=2023-11-27 13:39:00.0, currentDate=Mon Nov 27 13:39:01 CST 2023 2023-11-27 13:39:01.110 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-27 13:38:00.0, nextRunTime=2023-11-27 13:39:00.0} 2023-11-27 13:39:01.112 INFO 10 --- [pool-ocp-schedules-17,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2023-11-27 13:39:01.112 INFO 10 --- [pool-ocp-schedules-17,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2023-11-27 13:39:01.112 INFO 10 --- [pool-ocp-schedules-17,,] c.o.o.b.spring.BootstrapRunListener : set system property ocp.version to 4.0.3 2023-11-27 13:39:01.112 INFO 10 --- [pool-ocp-schedules-17,,] c.o.o.b.spring.BootstrapRunListener : set systemProperties as the first PropertySource 2023-11-27 13:39:01.112 INFO 10 --- [pool-ocp-schedules-17,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.0.3 2023-11-27 13:39:01.128 WARN 10 --- [pool-ocp-schedules-17,,] ironmentPostProcessorApplicationListener : Your Okta Issuer URL is missing. You can copy your domain from the Okta Developer Console. Follow these instructions to find it: https://bit.ly/finding-okta-domain To fix this add the `okta.oauth2.issuer` property to your application environments. 2023-11-27 13:39:01.128 INFO 10 --- [pool-ocp-schedules-17,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2023-11-27 13:39:01.161 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-11-27 13:39:01.161 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:39:01.161 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-11-27 13:39:01.175 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000061, operation:EXECUTE, state: RUNNING 2023-11-27 13:39:01.178 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:38:30.0, nextRun=2023-11-27 13:39:00.0, currentDate=Mon Nov 27 13:39:01 CST 2023 2023-11-27 13:39:01.178 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-11-27 13:38:30.0, nextRunTime=2023-11-27 13:39:00.0} 2023-11-27 13:39:01.182 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-11-27 13:39:01.182 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:39:01.182 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-11-27 13:39:01.186 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000061, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-11-27T13:39:01.172+08:00, endTime=null} 2023-11-27 13:39:01.216 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:38:00.0, nextRun=2023-11-27 13:39:00.0, currentDate=Mon Nov 27 13:39:01 CST 2023 2023-11-27 13:39:01.217 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=1000002, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-27 13:38:00.0, nextRunTime=2023-11-27 13:39:00.0} 2023-11-27 13:39:01.238 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-11-27 13:39:01.238 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:39:01.238 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=1000002, state=RUNNING, operation=EXECUTE} 2023-11-27 13:39:01.295 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000062, operation:EXECUTE, state: RUNNING 2023-11-27 13:39:01.299 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000062, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-11-27T13:39:01.255+08:00, endTime=null} 2023-11-27 13:39:01.331 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,eb5b0fe46713] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000061, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000051, task_operation=execute, latest_execution_start_time=2023-11-27T13:39:01.172+08:00, sub_task_instance_id=10000061}, listMap={}}, executor=10.120.0.1 2023-11-27 13:39:01.341 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,88e3f66e7ca2] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000062, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000052, task_operation=execute, latest_execution_start_time=2023-11-27T13:39:01.255+08:00, sub_task_instance_id=10000062}, listMap={}}, executor=10.120.0.1 2023-11-27 13:39:01.355 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=6 2023-11-27 13:39:01.355 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-11-27 13:39:01.356 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=9 2023-11-27 13:39:01.358 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2023-11-27 13:39:01.368 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,eb5b0fe46713] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000061, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:39:01.370 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:39:01.372 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2023-11-27 13:39:01.372 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2023-11-27 13:39:01.373 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:39:01.408 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,88e3f66e7ca2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-11-27 13:39:01.409 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,88e3f66e7ca2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster darenOB, trans stat result is empty. 2023-11-27 13:39:01.417 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,88e3f66e7ca2] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000062, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:39:01.487 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.12:62889/api/v1/log/alarms, params:null 2023-11-27 13:39:01.488 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.88:62889/api/v1/log/alarms, params:null 2023-11-27 13:39:01.492 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-11-27 13:39:01.500 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-11-27 13:39:01.507 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-11-27 13:39:01.515 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.92:62889/api/v1/log/alarms, params:null 2023-11-27 13:39:01.525 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-11-27 13:39:01.530 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:39:01.536 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.13:62889/api/v1/log/alarms, params:null 2023-11-27 13:39:01.537 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:39:01.558 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.11:62889/api/v1/log/alarms, params:null 2023-11-27 13:39:01.656 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.3:62889/api/v1/log/alarms, params:null 2023-11-27 13:39:01.662 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.97:62889/api/v1/log/alarms, params:null 2023-11-27 13:39:01.824 INFO 10 --- [pool-overview-container-4,,510bceea0b25] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:39:01.932 INFO 10 --- [pool-ocp-schedules-17,,] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2023-11-27 13:39:01.984 INFO 10 --- [pool-scanner--11823-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=88 2023-11-27 13:39:02.456 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000051, status:SUCCESS 2023-11-27 13:39:02.458 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000051, status:SUCCESS 2023-11-27 13:39:02.477 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000063, operation:EXECUTE, state: RUNNING 2023-11-27 13:39:02.502 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000063, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-11-27T13:39:02.472+08:00, endTime=null} 2023-11-27 13:39:02.508 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000052, status:SUCCESS 2023-11-27 13:39:02.514 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000052, status:SUCCESS 2023-11-27 13:39:02.547 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000064, operation:EXECUTE, state: RUNNING 2023-11-27 13:39:02.567 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,44d027abac50] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000063, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000053, task_operation=execute, latest_execution_start_time=2023-11-27T13:39:02.466+08:00, sub_task_instance_id=10000063}, listMap={}}, executor=10.120.0.1 2023-11-27 13:39:02.570 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000064, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-11-27T13:39:02.543+08:00, endTime=null} 2023-11-27 13:39:02.610 WARN 10 --- [pool-ocp-schedules-17,,] ironmentPostProcessorApplicationListener : Your Okta Issuer URL is missing. You can copy your domain from the Okta Developer Console. Follow these instructions to find it: https://bit.ly/finding-okta-domain To fix this add the `okta.oauth2.issuer` property to your application environments. 2023-11-27 13:39:02.611 INFO 10 --- [pool-ocp-schedules-17,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT2.596231611S 2023-11-27 13:39:02.612 INFO 10 --- [pool-ocp-schedules-17,,] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2023-11-27 13:39:02.744 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000065, operation:EXECUTE, state: RUNNING 2023-11-27 13:39:02.748 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000065, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-11-27T13:39:02.734+08:00, endTime=null} 2023-11-27 13:39:02.752 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,44d027abac50] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obcluster, sql count is 0. 2023-11-27 13:39:02.753 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,44d027abac50] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, result is empty. 2023-11-27 13:39:02.759 INFO 10 --- [pool-ocp-schedules-17,,] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2023-11-27 13:39:02.765 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,bb2d0d661809] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000064, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000054, task_operation=execute, latest_execution_start_time=2023-11-27T13:39:02.542+08:00, sub_task_instance_id=10000064}, listMap={}}, executor=10.120.0.1 2023-11-27 13:39:02.765 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,bb2d0d661809] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-11-27 13:39:02.806 INFO 10 --- [pool-915-thread-1,554481c3fd1b426b,782acf655606] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:darenOB, tenant:sys 2023-11-27 13:39:02.807 INFO 10 --- [pool-915-thread-2,554481c3fd1b426b,60e5ae8d0e2d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:darenOB, tenant:daren 2023-11-27 13:39:02.819 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b8f50992bce7] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000065, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000055, task_operation=execute, latest_execution_start_time=2023-11-27T13:39:02.733+08:00, sub_task_instance_id=10000065}, listMap={}}, executor=10.120.0.1 2023-11-27 13:39:02.899 INFO 10 --- [pool-915-thread-1,554481c3fd1b426b,782acf655606] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:02.905 INFO 10 --- [pool-915-thread-1,554481c3fd1b426b,782acf655606] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:02.905 INFO 10 --- [pool-915-thread-1,554481c3fd1b426b,782acf655606] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:02.908 INFO 10 --- [pool-915-thread-1,554481c3fd1b426b,782acf655606] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:02.908 INFO 10 --- [pool-915-thread-2,554481c3fd1b426b,60e5ae8d0e2d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:02.919 INFO 10 --- [pool-915-thread-2,554481c3fd1b426b,60e5ae8d0e2d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:02.920 INFO 10 --- [pool-915-thread-2,554481c3fd1b426b,60e5ae8d0e2d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:02.921 INFO 10 --- [pool-915-thread-2,554481c3fd1b426b,60e5ae8d0e2d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:02.921 INFO 10 --- [pool-915-thread-1,554481c3fd1b426b,782acf655606] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:02.921 INFO 10 --- [pool-915-thread-1,554481c3fd1b426b,782acf655606] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:02.923 INFO 10 --- [pool-915-thread-1,554481c3fd1b426b,782acf655606] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-11-27 13:39:02.925 INFO 10 --- [pool-915-thread-2,554481c3fd1b426b,60e5ae8d0e2d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:02.925 INFO 10 --- [pool-915-thread-2,554481c3fd1b426b,60e5ae8d0e2d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:02.926 INFO 10 --- [pool-915-thread-2,554481c3fd1b426b,60e5ae8d0e2d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-11-27 13:39:02.949 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b8f50992bce7] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000065, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:39:02.961 INFO 10 --- [pool-915-thread-2,554481c3fd1b426b,60e5ae8d0e2d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1701021600396785720, broadcast scn:1701021600396785720, saved frozen scn:1701021600396785720 2023-11-27 13:39:02.967 INFO 10 --- [pool-915-thread-2,554481c3fd1b426b,60e5ae8d0e2d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:darenOB, tenant:daren 2023-11-27 13:39:02.970 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:39:02.969597+08:00, lastProcessTime=2023-11-27T13:38:52.810744+08:00 2023-11-27 13:39:02.978 INFO 10 --- [pool-915-thread-1,554481c3fd1b426b,782acf655606] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1701021601748952808, broadcast scn:1701021601748952808, saved frozen scn:1701021601748952808 2023-11-27 13:39:02.979 INFO 10 --- [pool-915-thread-1,554481c3fd1b426b,782acf655606] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:darenOB, tenant:sys 2023-11-27 13:39:02.983 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,bb2d0d661809] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-11-27 13:39:02.996 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,bb2d0d661809] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000064, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:39:03.095 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:39:02.969597+08:00 2023-11-27 13:39:03.182 INFO 10 --- [pool-ocp-schedules-17,,] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}, BootstrapPropertySource {name='bootstrapProperties-ocp-composite'}, BootstrapPropertySource {name='bootstrapProperties-application-composite'}, BootstrapPropertySource {name='bootstrapProperties-application-default'}] 2023-11-27 13:39:03.229 INFO 10 --- [pool-ocp-schedules-17,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2023-11-27 13:39:03.229 INFO 10 --- [pool-ocp-schedules-17,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2023-11-27 13:39:03.229 INFO 10 --- [pool-ocp-schedules-17,,] c.o.o.b.spring.BootstrapRunListener : set system property ocp.version to 4.0.3 2023-11-27 13:39:03.229 INFO 10 --- [pool-ocp-schedules-17,,] c.o.o.b.spring.BootstrapRunListener : set systemProperties as the first PropertySource 2023-11-27 13:39:03.229 INFO 10 --- [pool-ocp-schedules-17,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.0.3 2023-11-27 13:39:03.230 INFO 10 --- [pool-ocp-schedules-17,,] o.s.boot.SpringApplication : The following 2 profiles are active: "composite", "default" 2023-11-27 13:39:03.256 INFO 10 --- [pool-ocp-schedules-17,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2023-11-27 13:39:03.387 INFO 10 --- [pool-ocp-schedules-17,,] o.s.boot.SpringApplication : Started application in 3.377 seconds (JVM running for 359.526) 2023-11-27 13:39:03.388 INFO 10 --- [pool-ocp-schedules-17,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT3.377499071S 2023-11-27 13:39:03.422 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,44d027abac50] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:03.433 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,44d027abac50] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:03.433 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,44d027abac50] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:03.434 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,44d027abac50] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:03.443 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,44d027abac50] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:03.443 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,44d027abac50] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:03.443 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,44d027abac50] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-11-27 13:39:03.610 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,44d027abac50] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for darenOB, save 1 to SqlAggregateData 2023-11-27 13:39:03.659 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,44d027abac50] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000063, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:39:03.707 INFO 10 --- [pool-ocp-schedules-17,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] driver class=com.oceanbase.jdbc.Driver 2023-11-27 13:39:03.707 INFO 10 --- [pool-ocp-schedules-17,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] username=meta_user@meta_tenant#obcluster 2023-11-27 13:39:03.707 INFO 10 --- [pool-ocp-schedules-17,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] jdbc url=jdbc:oceanbase://133.197.206.97:2883/meta_database?useUnicode=true&characterEncoding=UTF8&connectTimeout=2000&socketTimeout=30000&rewriteBatchedStatements=true 2023-11-27 13:39:03.707 INFO 10 --- [pool-ocp-schedules-17,,] c.o.o.b.spring.DataSourceInterceptor : dataSource metaDataSource already initialized. skipped. class: com.oceanbase.ocp.server.config.wrapper.MetadbDataSourceWrapper 2023-11-27 13:39:03.814 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000053, status:SUCCESS 2023-11-27 13:39:03.816 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000053, status:SUCCESS 2023-11-27 13:39:03.895 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000054, status:SUCCESS 2023-11-27 13:39:03.904 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000054, status:SUCCESS 2023-11-27 13:39:04.022 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000055, status:SUCCESS 2023-11-27 13:39:04.028 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000055, status:SUCCESS 2023-11-27 13:39:04.097 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:39:04.104 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:39:04.153 INFO 10 --- [pool-ocp-schedules-12,,240a3a1c9503] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=585625224722984960 2023-11-27 13:39:04.240 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:39:04.317 INFO 10 --- [pool-ocp-schedules-17,,] c.o.ocp.core.property.PropertyService : Refresh config properties done 2023-11-27 13:39:04.440 INFO 10 --- [pool-ocp-schedules-15,,692c97ed5d65] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:39:04.877 INFO 10 --- [alarm-detect-8,,4de40ac33a3f] c.o.o.s.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans, startId: 348000001, data is empty. 2023-11-27 13:39:04.906 INFO 10 --- [alarm-detect-8,,4de40ac33a3f] c.o.o.s.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans, startId: 348000001, data is empty. 2023-11-27 13:39:05.001 INFO 10 --- [alarm-detect-9,,74b8fe1b21ba] c.o.o.s.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist, startId: 348000001, data is empty. 2023-11-27 13:39:05.012 INFO 10 --- [alarm-detect-7,,2d4259a891d9] c.o.o.s.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist, startId: 348000001, data is empty. 2023-11-27 13:39:05.014 INFO 10 --- [alarm-detect-9,,74b8fe1b21ba] c.o.o.s.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist, startId: 348000001, data is empty. 2023-11-27 13:39:05.017 INFO 10 --- [alarm-detect-7,,2d4259a891d9] c.o.o.s.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist, startId: 348000001, data is empty. 2023-11-27 13:39:05.019 INFO 10 --- [alarm-detect-4,,d706619d5cd3] c.o.o.s.a.core.detect.MetricDetector : alarmType host_unavailable detect time range 1701063474 ~ 1701063534 matched, alarm-level: 1, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={host_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=host_available, match=true, value=0.0, values={1701063480=0.0}, valueCount=1, labels={app=HOST, host=133.197.206.92, host_name=localhost.localdomain, ob_cluster_id=, ob_cluster_name=, svr_ip=133.197.206.92})]}, matched=true, message=null, level=1) 2023-11-27 13:39:05.019 INFO 10 --- [alarm-detect-4,,d706619d5cd3] c.o.o.s.a.core.detect.MetricDetector : alarmType host_unavailable detect time range 1701063474 ~ 1701063534 matched, alarm-level: 1, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={host_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=host_available, match=true, value=0.0, values={1701063480=0.0}, valueCount=1, labels={app=HOST, host=133.197.206.97, host_name=localhost.localdomain, ob_cluster_id=1, ob_cluster_name=obcluster, svr_ip=133.197.206.97})]}, matched=true, message=null, level=1) 2023-11-27 13:39:05.019 INFO 10 --- [alarm-detect-4,,d706619d5cd3] c.o.o.s.a.core.detect.MetricDetector : alarmType host_unavailable detect time range 1701063474 ~ 1701063534 matched, alarm-level: 1, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={host_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=host_available, match=true, value=0.0, values={1701063480=0.0}, valueCount=1, labels={app=HOST, host=133.197.206.88, host_name=localhost.localdomain, ob_cluster_id=, ob_cluster_name=, svr_ip=133.197.206.88})]}, matched=true, message=null, level=1) 2023-11-27 13:39:05.019 INFO 10 --- [alarm-detect-4,,d706619d5cd3] c.o.o.s.a.core.detect.MetricDetector : alarmType host_unavailable detect time range 1701063474 ~ 1701063534 matched, alarm-level: 1, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={host_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=host_available, match=true, value=0.0, values={1701063480=0.0}, valueCount=1, labels={app=HOST, host=133.197.204.3, host_name=localhost.localdomain, ob_cluster_id=10, ob_cluster_name=darenOB, svr_ip=133.197.204.3})]}, matched=true, message=null, level=1) 2023-11-27 13:39:05.031 INFO 10 --- [alarm-detect-13,,02796e2ab9ab] c.o.o.s.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist, startId: 348000001, data is empty. 2023-11-27 13:39:05.043 INFO 10 --- [alarm-detect-5,,b3873a377e6a] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_cluster_exists_inactive_server detect time range 1701063525 ~ 1701063535 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={inactive_server_count > 0=[ExpressionEvaluateResult(matchValue=0.0, metric=inactive_server_count, match=true, value=1.0, values={1701063530=1.0}, valueCount=1, labels={app=OB, ob_cluster_id=10, ob_cluster_name=darenOB, server_ips=133.197.204.3})]}, matched=true, message=null, level=2) 2023-11-27 13:39:05.057 INFO 10 --- [alarm-detect-13,,02796e2ab9ab] c.o.o.s.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist, startId: 348000001, data is empty. 2023-11-27 13:39:05.081 INFO 10 --- [alarm-detect-6,,f57844a9cd8c] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063474 ~ 1701063534 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063480=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/obproxy, exporter_addr=http://133.197.204.3:62889/metrics/obproxy, exporter_type=OB_PROXY, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:39:05.082 INFO 10 --- [alarm-detect-6,,f57844a9cd8c] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063474 ~ 1701063534 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063480=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/ob/basic, exporter_addr=http://133.197.206.97:62889/metrics/ob/basic, exporter_type=OB_CLUSTER, host=133.197.206.97, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.206.97})]}, matched=true, message=null, level=3) 2023-11-27 13:39:05.082 INFO 10 --- [alarm-detect-6,,f57844a9cd8c] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063474 ~ 1701063534 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063480=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/ob, exporter_addr=http://133.197.206.97:62889/metrics/node/ob, exporter_type=OB_CLUSTER, host=133.197.206.97, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.206.97})]}, matched=true, message=null, level=3) 2023-11-27 13:39:05.082 INFO 10 --- [alarm-detect-6,,f57844a9cd8c] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063474 ~ 1701063534 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063480=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://133.197.206.92:62889/metrics/stat, exporter_type=HOST_MONITOR, host=133.197.206.92, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.206.92})]}, matched=true, message=null, level=3) 2023-11-27 13:39:05.082 INFO 10 --- [alarm-detect-6,,f57844a9cd8c] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063474 ~ 1701063534 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063480=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/ob/basic, exporter_addr=http://133.197.204.3:62889/metrics/ob/basic, exporter_type=OB_CLUSTER, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:39:05.082 INFO 10 --- [alarm-detect-6,,f57844a9cd8c] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063474 ~ 1701063534 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063480=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://133.197.204.3:62888/metrics/stat, exporter_type=HOST_MONITOR, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:39:05.082 INFO 10 --- [alarm-detect-6,,f57844a9cd8c] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063474 ~ 1701063534 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063480=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/ob/extra, exporter_addr=http://133.197.206.97:62889/metrics/ob/extra, exporter_type=OB_CLUSTER, host=133.197.206.97, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.206.97})]}, matched=true, message=null, level=3) 2023-11-27 13:39:05.082 INFO 10 --- [alarm-detect-6,,f57844a9cd8c] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063474 ~ 1701063534 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063480=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://133.197.206.97:62888/metrics/stat, exporter_type=HOST_MONITOR, host=133.197.206.97, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.206.97})]}, matched=true, message=null, level=3) 2023-11-27 13:39:05.083 INFO 10 --- [alarm-detect-6,,f57844a9cd8c] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063474 ~ 1701063534 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063480=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://133.197.206.92:62888/metrics/stat, exporter_type=HOST_MONITOR, host=133.197.206.92, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.206.92})]}, matched=true, message=null, level=3) 2023-11-27 13:39:05.083 INFO 10 --- [alarm-detect-6,,f57844a9cd8c] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063474 ~ 1701063534 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063480=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/host, exporter_addr=http://133.197.204.3:62889/metrics/node/host, exporter_type=HOST_MONITOR, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:39:05.083 INFO 10 --- [alarm-detect-6,,f57844a9cd8c] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063474 ~ 1701063534 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063480=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://133.197.204.3:62889/metrics/stat, exporter_type=HOST_MONITOR, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:39:05.083 INFO 10 --- [alarm-detect-6,,f57844a9cd8c] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063474 ~ 1701063534 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063480=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://133.197.206.88:62888/metrics/stat, exporter_type=HOST_MONITOR, host=133.197.206.88, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.206.88})]}, matched=true, message=null, level=3) 2023-11-27 13:39:05.083 INFO 10 --- [alarm-detect-6,,f57844a9cd8c] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063474 ~ 1701063534 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063480=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/host, exporter_addr=http://133.197.206.92:62889/metrics/node/host, exporter_type=HOST_MONITOR, host=133.197.206.92, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.206.92})]}, matched=true, message=null, level=3) 2023-11-27 13:39:05.083 INFO 10 --- [alarm-detect-6,,f57844a9cd8c] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063474 ~ 1701063534 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063480=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://133.197.206.97:62889/metrics/stat, exporter_type=HOST_MONITOR, host=133.197.206.97, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.206.97})]}, matched=true, message=null, level=3) 2023-11-27 13:39:05.083 INFO 10 --- [alarm-detect-6,,f57844a9cd8c] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063474 ~ 1701063534 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063480=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/host, exporter_addr=http://133.197.206.88:62889/metrics/node/host, exporter_type=HOST_MONITOR, host=133.197.206.88, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.206.88})]}, matched=true, message=null, level=3) 2023-11-27 13:39:05.083 INFO 10 --- [alarm-detect-6,,f57844a9cd8c] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063474 ~ 1701063534 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063480=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/ob, exporter_addr=http://133.197.204.3:62889/metrics/node/ob, exporter_type=OB_CLUSTER, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:39:05.083 INFO 10 --- [alarm-detect-6,,f57844a9cd8c] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063474 ~ 1701063534 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063480=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/ob/extra, exporter_addr=http://133.197.204.3:62889/metrics/ob/extra, exporter_type=OB_CLUSTER, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:39:05.083 INFO 10 --- [alarm-detect-6,,f57844a9cd8c] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063474 ~ 1701063534 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063480=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/host, exporter_addr=http://133.197.206.97:62889/metrics/node/host, exporter_type=HOST_MONITOR, host=133.197.206.97, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.206.97})]}, matched=true, message=null, level=3) 2023-11-27 13:39:05.084 INFO 10 --- [alarm-detect-6,,f57844a9cd8c] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063474 ~ 1701063534 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063480=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/obproxy, exporter_addr=http://133.197.204.3:62889/metrics/node/obproxy, exporter_type=OB_PROXY, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:39:05.084 INFO 10 --- [alarm-detect-6,,f57844a9cd8c] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063474 ~ 1701063534 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063480=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://133.197.206.88:62889/metrics/stat, exporter_type=HOST_MONITOR, host=133.197.206.88, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.206.88})]}, matched=true, message=null, level=3) 2023-11-27 13:39:05.084 INFO 10 --- [alarm-detect-6,,f57844a9cd8c] c.o.o.s.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=monitor_exporter_unavaliable, target=alarm_template_id=0:host=133.197.206.92:exporter=/metrics/stat-3, currentState=Pending 2023-11-27 13:39:05.084 INFO 10 --- [alarm-detect-6,,f57844a9cd8c] c.o.o.s.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=monitor_exporter_unavaliable, target=alarm_template_id=0:host=133.197.206.97:exporter=/metrics/ob/extra-3, currentState=Pending 2023-11-27 13:39:05.085 INFO 10 --- [alarm-detect-6,,f57844a9cd8c] c.o.o.s.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=monitor_exporter_unavaliable, target=alarm_template_id=0:host=133.197.206.97:exporter=/metrics/stat-3, currentState=Pending 2023-11-27 13:39:05.085 INFO 10 --- [alarm-detect-6,,f57844a9cd8c] c.o.o.s.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=monitor_exporter_unavaliable, target=alarm_template_id=0:host=133.197.206.88:exporter=/metrics/stat-3, currentState=Pending 2023-11-27 13:39:05.104 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063545, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:05.112 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063545, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:05.112 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063545, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:05.113 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063545, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:05.114 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063545, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:05.144 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:39:05.191 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:39:05.271 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:39:05.732 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:39:05.926 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:38:06.0, nextRun=2023-11-27 13:39:05.0, currentDate=Mon Nov 27 13:39:05 CST 2023 2023-11-27 13:39:05.927 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-11-27 13:38:06.0, nextRunTime=2023-11-27 13:39:05.0} 2023-11-27 13:39:05.953 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-11-27 13:39:05.954 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:39:05.954 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-11-27 13:39:06.289 INFO 10 --- [http-nio-0.0.0.0-8080-exec-7,b64325cf10454a98,3c491aad0988] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.12, traceId=b64325cf10454a98, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:39:06.296 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:39:06.326 INFO 10 --- [http-nio-0.0.0.0-8080-exec-7,b64325cf10454a98,3c491aad0988] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.12, traceId=b64325cf10454a98, duration=39 ms] 2023-11-27 13:39:06.876 INFO 10 --- [pool-subtask-coordinator4,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000066, operation:EXECUTE, state: RUNNING 2023-11-27 13:39:06.885 INFO 10 --- [pool-subtask-coordinator4,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000066, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-11-27T13:39:06.873+08:00, endTime=null} 2023-11-27 13:39:06.909 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,2f1d5c73a153] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000066, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000056, task_operation=execute, latest_execution_start_time=2023-11-27T13:39:06.872+08:00, sub_task_instance_id=10000066}, listMap={}}, executor=10.120.0.1 2023-11-27 13:39:06.955 INFO 10 --- [http-nio-0.0.0.0-8080-exec-6,3610fc5e6315471b,d4e4f4a407f8] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.12, traceId=3610fc5e6315471b, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:39:07.008 INFO 10 --- [pool-Check obproxy1,554481c3fd1b426b,aae88d8e56a7] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-11-27 13:39:07.014 WARN 10 --- [pool-Check obproxy1,554481c3fd1b426b,aae88d8e56a7] c.oceanbase.ocp.odpsdk.OdpJdbcOperator : Failed to check odp connection Connection {driverClassName ='com.oceanbase.jdbc.Driver', connectionProperties ='[user=root@proxysys, url=jdbc:oceanbase://133.197.204.3:2883/?connectProxy=true]'}, error is Failed to obtain JDBC Connection; nested exception is java.sql.SQLNonTransientConnectionException: Could not connect to HostAddress{host='133.197.204.3', port=2883}. Connection refused (Connection refused) 2023-11-27 13:39:07.014 WARN 10 --- [pool-Check obproxy1,554481c3fd1b426b,aae88d8e56a7] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to connect obproxy server, maybe caused by odp process down, wrong password, or ocp address not in private ip list and odp parameter skip_proxy_sys_private_check is not true 2023-11-27 13:39:07.014 ERROR 10 --- [pool-Check obproxy1,554481c3fd1b426b,aae88d8e56a7] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.UnexpectedException occurred with code error.obproxy.connection.error, and args [133.197.204.3:2883] 2023-11-27 13:39:07.016 INFO 10 --- [http-nio-0.0.0.0-8080-exec-6,3610fc5e6315471b,d4e4f4a407f8] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.12, traceId=3610fc5e6315471b, duration=62 ms] 2023-11-27 13:39:07.016 INFO 10 --- [pool-Check obproxy5,554481c3fd1b426b,9cf50ffc29cd] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-11-27 13:39:07.057 INFO 10 --- [pool-Check obproxy3,554481c3fd1b426b,3b7b1abca840] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-11-27 13:39:07.060 WARN 10 --- [pool-Check obproxy1,554481c3fd1b426b,aae88d8e56a7] c.oceanbase.ocp.odpsdk.OdpJdbcOperator : Failed to check odp connection Connection {driverClassName ='com.oceanbase.jdbc.Driver', connectionProperties ='[user=root@proxysys, url=jdbc:oceanbase://133.197.204.3:2883/?connectProxy=true]'}, error is Failed to obtain JDBC Connection; nested exception is java.sql.SQLNonTransientConnectionException: Could not connect to HostAddress{host='133.197.204.3', port=2883}. Connection refused (Connection refused) 2023-11-27 13:39:07.060 WARN 10 --- [pool-Check obproxy1,554481c3fd1b426b,aae88d8e56a7] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to connect obproxy server, maybe caused by odp process down, wrong password, or ocp address not in private ip list and odp parameter skip_proxy_sys_private_check is not true 2023-11-27 13:39:07.060 ERROR 10 --- [pool-Check obproxy1,554481c3fd1b426b,aae88d8e56a7] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.UnexpectedException occurred with code error.obproxy.connection.error, and args [133.197.204.3:2883] 2023-11-27 13:39:07.063 INFO 10 --- [pool-Check obproxy5,554481c3fd1b426b,9cf50ffc29cd] c.o.o.s.o.obproxy.ObproxyService : check obproxy 12 success, update last active time to now 2023-11-27 13:39:07.078 INFO 10 --- [pool-Check obproxy2,554481c3fd1b426b,a1835636d43c] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-11-27 13:39:07.105 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:39:07.110 INFO 10 --- [pool-Check obproxy2,554481c3fd1b426b,a1835636d43c] c.o.o.s.o.obproxy.ObproxyService : check obproxy 10 success, update last active time to now 2023-11-27 13:39:07.114 INFO 10 --- [pool-Check obproxy1,554481c3fd1b426b,aae88d8e56a7] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 failed, will not update last active time com.oceanbase.ocp.core.exception.UnexpectedException: [OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=OBPROXY_CONNECTION_ERROR, args=133.197.204.3:2883 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_312] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_312] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_312] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_312] at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.util.ExceptionUtils.throwException(ExceptionUtils.java:90) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.util.ExceptionUtils.unExpected(ExceptionUtils.java:77) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.service.operation.obproxy.SystemOdpOperatorFactory.createOdpOperator(SystemOdpOperatorFactory.java:105) ~[ocp-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.service.operation.obproxy.SystemOdpOperatorFactory.tryToCreateOdpOperator(SystemOdpOperatorFactory.java:86) ~[ocp-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.service.operation.obproxy.ObproxyService.lambda$checkObproxyServers$20(ObproxyService.java:517) ~[ocp-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] 2023-11-27 13:39:07.116 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:39:07.121 INFO 10 --- [pool-Check obproxy3,554481c3fd1b426b,3b7b1abca840] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-11-27 13:39:07.127 INFO 10 --- [pool-Check obproxy4,554481c3fd1b426b,25d99f97f70f] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-11-27 13:39:07.159 INFO 10 --- [pool-Check obproxy4,554481c3fd1b426b,25d99f97f70f] c.o.o.s.o.obproxy.ObproxyService : check obproxy 11 success, update last active time to now 2023-11-27 13:39:07.175 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,2f1d5c73a153] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000066, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:39:07.271 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000056, status:SUCCESS 2023-11-27 13:39:07.272 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000056, status:SUCCESS 2023-11-27 13:39:09.016 INFO 10 --- [pool-prometheus-exporter-check2,,86a6d982d625] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/stat, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:39:09.016 WARN 10 --- [pool-prometheus-exporter-check2,,86a6d982d625] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/stat is inactive 2023-11-27 13:39:09.018 INFO 10 --- [pool-prometheus-exporter-check2,,86a6d982d625] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62888/metrics/stat, message=java.net.ConnectException: Connection refused: /133.197.204.3:62888 2023-11-27 13:39:09.018 WARN 10 --- [pool-prometheus-exporter-check2,,86a6d982d625] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62888/metrics/stat is inactive 2023-11-27 13:39:09.019 INFO 10 --- [pool-prometheus-exporter-check2,,86a6d982d625] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/node/host, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:39:09.019 WARN 10 --- [pool-prometheus-exporter-check2,,86a6d982d625] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/node/host is inactive 2023-11-27 13:39:09.024 INFO 10 --- [pool-prometheus-exporter-check2,,86a6d982d625] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/obproxy, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:39:09.024 WARN 10 --- [pool-prometheus-exporter-check2,,86a6d982d625] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/obproxy is inactive 2023-11-27 13:39:09.025 INFO 10 --- [pool-prometheus-exporter-check2,,86a6d982d625] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/node/obproxy, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:39:09.025 WARN 10 --- [pool-prometheus-exporter-check2,,86a6d982d625] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/node/obproxy is inactive 2023-11-27 13:39:09.026 INFO 10 --- [pool-prometheus-exporter-check2,,86a6d982d625] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/ob/extra, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:39:09.026 WARN 10 --- [pool-prometheus-exporter-check2,,86a6d982d625] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/ob/extra is inactive 2023-11-27 13:39:09.030 INFO 10 --- [pool-prometheus-exporter-check2,,86a6d982d625] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/node/ob, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:39:09.030 WARN 10 --- [pool-prometheus-exporter-check2,,86a6d982d625] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/node/ob is inactive 2023-11-27 13:39:09.032 INFO 10 --- [pool-prometheus-exporter-check2,,86a6d982d625] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/ob/basic, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:39:09.032 WARN 10 --- [pool-prometheus-exporter-check2,,86a6d982d625] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/ob/basic is inactive 2023-11-27 13:39:09.627 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.120.0.1:8080 2023-11-27 13:39:09.942 INFO 10 --- [http-nio-0.0.0.0-8080-exec-1,b2ec2bd040504cad,520fcfc81989] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.10, traceId=b2ec2bd040504cad, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:39:09.951 INFO 10 --- [http-nio-0.0.0.0-8080-exec-1,b2ec2bd040504cad,520fcfc81989] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.10, traceId=b2ec2bd040504cad, duration=10 ms] 2023-11-27 13:39:10.105 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063550, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:10.105 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063550, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:10.105 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063550, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:10.105 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063550, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:10.115 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063550, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:10.116 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:39:10.119 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:39:11.799 INFO 10 --- [pool-overview-container-7,,6738f778a0a0] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:39:11.879 INFO 10 --- [pool-scanner--11954-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=88 2023-11-27 13:39:11.966 INFO 10 --- [http-nio-0.0.0.0-8080-exec-3,f63a03a7ec514dab,249fe91712ca] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.11, traceId=f63a03a7ec514dab, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:39:11.996 INFO 10 --- [http-nio-0.0.0.0-8080-exec-3,f63a03a7ec514dab,249fe91712ca] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.11, traceId=f63a03a7ec514dab, duration=30 ms] 2023-11-27 13:39:12.125 INFO 10 --- [http-nio-0.0.0.0-8080-exec-5,d7fcaf1776cd4a73,06b370d1a96c] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.10, traceId=d7fcaf1776cd4a73, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:39:12.131 INFO 10 --- [http-nio-0.0.0.0-8080-exec-5,d7fcaf1776cd4a73,06b370d1a96c] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.10, traceId=d7fcaf1776cd4a73, duration=6 ms] 2023-11-27 13:39:12.779 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:39:12.777842+08:00, lastProcessTime=2023-11-27T13:39:02.969597+08:00 2023-11-27 13:39:12.931 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:39:12.777842+08:00 2023-11-27 13:39:13.122 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:39:13.126 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:39:13.219 INFO 10 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1701063480, nodeCount=1000, affectRows=1000 2023-11-27 13:39:13.223 INFO 10 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1701063480, nodeCount=1000, validCount=1000 2023-11-27 13:39:13.250 INFO 10 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1701063480, nodeCount=707, affectRows=707 2023-11-27 13:39:13.253 INFO 10 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1701063480, nodeCount=707, validCount=707 2023-11-27 13:39:13.871 INFO 10 --- [http-nio-0.0.0.0-8080-exec-7,dcfed37e77ba4280,88a6ff0318f2] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.13, traceId=dcfed37e77ba4280, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:39:13.892 INFO 10 --- [http-nio-0.0.0.0-8080-exec-7,dcfed37e77ba4280,88a6ff0318f2] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.13, traceId=dcfed37e77ba4280, duration=21 ms] 2023-11-27 13:39:14.235 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:39:14.392 INFO 10 --- [pool-ocp-schedules-15,,b97297e0f545] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-11-27 13:39:14.427 INFO 10 --- [pool-Check host status24,,f6a2110ee7ac] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=133.197.206.97, originalStatus=OFFLINE 2023-11-27 13:39:14.427 INFO 10 --- [pool-Check host status18,,45b644aa8a5b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=133.197.204.3, originalStatus=OFFLINE 2023-11-27 13:39:14.440 INFO 10 --- [pool-Check host status21,,8984b5560838] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=133.197.206.92, originalStatus=OFFLINE 2023-11-27 13:39:14.441 INFO 10 --- [pool-Check host status22,,b18a4b8381b8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=133.197.204.10, originalStatus=ONLINE 2023-11-27 13:39:14.441 INFO 10 --- [pool-Check host status23,,ae034093a873] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=133.197.204.11, originalStatus=ONLINE 2023-11-27 13:39:14.444 INFO 10 --- [pool-Check host status18,,45b644aa8a5b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:39:14.447 INFO 10 --- [pool-Check host status24,,f6a2110ee7ac] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:39:14.452 INFO 10 --- [pool-Check host status23,,ae034093a873] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-11-27 13:39:14.457 INFO 10 --- [pool-Check host status22,,b18a4b8381b8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-11-27 13:39:14.460 INFO 10 --- [pool-Check host status21,,8984b5560838] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:39:14.496 INFO 10 --- [pool-Check host status20,,0908082dfebe] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=6, ip=133.197.204.12, originalStatus=ONLINE 2023-11-27 13:39:14.496 INFO 10 --- [pool-Check host status28,,159ce6638647] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=133.197.204.13, originalStatus=ONLINE 2023-11-27 13:39:14.504 INFO 10 --- [pool-Check host status29,,9a8f5950771c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=9, ip=133.197.206.88, originalStatus=OFFLINE 2023-11-27 13:39:14.533 INFO 10 --- [pool-Check host status20,,0908082dfebe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2023-11-27 13:39:14.541 INFO 10 --- [pool-ocp-schedules-7,,559e0869affc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:39:14.544 INFO 10 --- [pool-Check host status22,,b18a4b8381b8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.10:62888/api/v1/time, params:null 2023-11-27 13:39:14.548 INFO 10 --- [pool-Check host status22,,b18a4b8381b8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=133.197.204.10, timestamp=2023-11-27T00:39:14.539633582-05:00 2023-11-27 13:39:14.549 INFO 10 --- [pool-Check host status18,,45b644aa8a5b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.3:62888/api/v1/time, params:null 2023-11-27 13:39:14.552 INFO 10 --- [pool-Check host status29,,9a8f5950771c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:39:14.552 INFO 10 --- [pool-Check host status22,,b18a4b8381b8] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-11-27T13:39:14.548+08:00 2023-11-27 13:39:14.559 WARN 10 --- [pool-Check host status18,,45b644aa8a5b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check failed. host=2, ip=133.197.204.3, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.204.3:62888 [/133.197.204.3] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.getHostTime(AgentExecutor.java:127) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.doCheckHostStatus(HostCheckManagerImpl.java:206) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostStatus(HostCheckManagerImpl.java:168) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.204.3:62888 [/133.197.204.3] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 24 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 33 common frames omitted 2023-11-27 13:39:14.568 INFO 10 --- [pool-Check host status28,,159ce6638647] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2023-11-27 13:39:14.573 INFO 10 --- [pool-Check host status18,,45b644aa8a5b] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [2:133.197.204.3], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:39:14.573 INFO 10 --- [pool-Check host status18,,45b644aa8a5b] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:39:14.580 INFO 10 --- [pool-Check host status22,,b18a4b8381b8] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 11 last available time 2023-11-27T13:39:14.548+08:00 and status AVAILABLE 2023-11-27 13:39:14.585 INFO 10 --- [pool-Check host status18,,45b644aa8a5b] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=2 2023-11-27 13:39:14.585 INFO 10 --- [pool-Check host status18,,45b644aa8a5b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2023-11-27 13:39:14.589 INFO 10 --- [pool-Check host status18,,45b644aa8a5b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:39:14.632 INFO 10 --- [pool-Check host status20,,0908082dfebe] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.12:62888/api/v1/time, params:null 2023-11-27 13:39:14.636 INFO 10 --- [pool-Check host status20,,0908082dfebe] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=6, ip=133.197.204.12, timestamp=2023-11-27T00:39:14.630447735-05:00 2023-11-27 13:39:14.640 INFO 10 --- [pool-Check host status20,,0908082dfebe] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=6, lastAvailableTime=2023-11-27T13:39:14.636+08:00 2023-11-27 13:39:14.645 INFO 10 --- [pool-Check host status20,,0908082dfebe] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 13 last available time 2023-11-27T13:39:14.636+08:00 and status AVAILABLE 2023-11-27 13:39:14.647 INFO 10 --- [pool-Check host status22,,b18a4b8381b8] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-11-27T13:39:14.644+08:00 2023-11-27 13:39:14.652 INFO 10 --- [pool-Check host status22,,b18a4b8381b8] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 11 last available time 2023-11-27T13:39:14.644+08:00 and status AVAILABLE 2023-11-27 13:39:14.663 INFO 10 --- [pool-Check host status20,,0908082dfebe] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=6, lastAvailableTime=2023-11-27T13:39:14.662+08:00 2023-11-27 13:39:14.666 INFO 10 --- [pool-Check host status20,,0908082dfebe] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 13 last available time 2023-11-27T13:39:14.662+08:00 and status AVAILABLE 2023-11-27 13:39:14.670 INFO 10 --- [pool-Check host status22,,b18a4b8381b8] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-11-27 13:39:14.670 INFO 10 --- [pool-Check host status21,,8984b5560838] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.92:62888/api/v1/time, params:null 2023-11-27 13:39:14.672 WARN 10 --- [pool-Check host status21,,8984b5560838] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check failed. host=3, ip=133.197.206.92, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.92:62888 [/133.197.206.92] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.getHostTime(AgentExecutor.java:127) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.doCheckHostStatus(HostCheckManagerImpl.java:206) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostStatus(HostCheckManagerImpl.java:168) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.92:62888 [/133.197.206.92] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 24 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 33 common frames omitted 2023-11-27 13:39:14.676 INFO 10 --- [pool-Check host status23,,ae034093a873] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.11:62888/api/v1/time, params:null 2023-11-27 13:39:14.677 INFO 10 --- [pool-Check host status20,,0908082dfebe] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=6 2023-11-27 13:39:14.681 INFO 10 --- [pool-Check host status23,,ae034093a873] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=133.197.204.11, timestamp=2023-11-27T00:39:14.676955464-05:00 2023-11-27 13:39:14.682 INFO 10 --- [pool-Check host status23,,ae034093a873] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-11-27T13:39:14.681+08:00 2023-11-27 13:39:14.685 INFO 10 --- [pool-Check host status21,,8984b5560838] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [3:133.197.206.92], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:39:14.685 INFO 10 --- [pool-Check host status21,,8984b5560838] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:39:14.686 INFO 10 --- [pool-Check host status24,,f6a2110ee7ac] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.97:62888/api/v1/time, params:null 2023-11-27 13:39:14.688 WARN 10 --- [pool-Check host status24,,f6a2110ee7ac] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check failed. host=1, ip=133.197.206.97, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.97:62888 [/133.197.206.97] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.getHostTime(AgentExecutor.java:127) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.doCheckHostStatus(HostCheckManagerImpl.java:206) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostStatus(HostCheckManagerImpl.java:168) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.97:62888 [/133.197.206.97] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 24 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 33 common frames omitted 2023-11-27 13:39:14.692 INFO 10 --- [pool-Check host status23,,ae034093a873] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 12 last available time 2023-11-27T13:39:14.681+08:00 and status AVAILABLE 2023-11-27 13:39:14.698 INFO 10 --- [pool-Check host status22,,b18a4b8381b8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-11-27 13:39:14.699 INFO 10 --- [pool-Check host status21,,8984b5560838] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=3 2023-11-27 13:39:14.699 INFO 10 --- [pool-Check host status21,,8984b5560838] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2023-11-27 13:39:14.701 INFO 10 --- [pool-Check host status24,,f6a2110ee7ac] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [1:133.197.206.97], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:39:14.701 INFO 10 --- [pool-Check host status24,,f6a2110ee7ac] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:39:14.708 INFO 10 --- [pool-Check host status20,,0908082dfebe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2023-11-27 13:39:14.709 INFO 10 --- [pool-Check host status21,,8984b5560838] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:39:14.711 INFO 10 --- [pool-Check host status24,,f6a2110ee7ac] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=1 2023-11-27 13:39:14.711 INFO 10 --- [pool-Check host status24,,f6a2110ee7ac] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2023-11-27 13:39:14.718 INFO 10 --- [pool-Check host status24,,f6a2110ee7ac] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:39:14.756 INFO 10 --- [pool-Check host status29,,9a8f5950771c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.88:62888/api/v1/time, params:null 2023-11-27 13:39:14.759 INFO 10 --- [pool-Check host status28,,159ce6638647] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.13:62888/api/v1/time, params:null 2023-11-27 13:39:14.760 WARN 10 --- [pool-Check host status29,,9a8f5950771c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check failed. host=9, ip=133.197.206.88, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.88:62888 [/133.197.206.88] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.getHostTime(AgentExecutor.java:127) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.doCheckHostStatus(HostCheckManagerImpl.java:206) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostStatus(HostCheckManagerImpl.java:168) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.88:62888 [/133.197.206.88] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 24 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 33 common frames omitted 2023-11-27 13:39:14.763 INFO 10 --- [pool-Check host status28,,159ce6638647] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=7, ip=133.197.204.13, timestamp=2023-11-27T00:39:14.744917749-05:00 2023-11-27 13:39:14.763 INFO 10 --- [pool-Check host status23,,ae034093a873] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-11-27T13:39:14.758+08:00 2023-11-27 13:39:14.765 INFO 10 --- [pool-Check host status28,,159ce6638647] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2023-11-27T13:39:14.763+08:00 2023-11-27 13:39:14.769 INFO 10 --- [pool-Check host status29,,9a8f5950771c] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [9:133.197.206.88], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:39:14.769 INFO 10 --- [pool-Check host status29,,9a8f5950771c] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:39:14.779 INFO 10 --- [pool-Check host status28,,159ce6638647] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 14 last available time 2023-11-27T13:39:14.763+08:00 and status AVAILABLE 2023-11-27 13:39:14.780 INFO 10 --- [pool-Check host status29,,9a8f5950771c] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=9 2023-11-27 13:39:14.780 INFO 10 --- [pool-Check host status29,,9a8f5950771c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=9 2023-11-27 13:39:14.784 INFO 10 --- [pool-Check host status23,,ae034093a873] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 12 last available time 2023-11-27T13:39:14.758+08:00 and status AVAILABLE 2023-11-27 13:39:14.786 INFO 10 --- [pool-Check host status29,,9a8f5950771c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:39:14.820 INFO 10 --- [pool-Check host status23,,ae034093a873] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-11-27 13:39:14.831 INFO 10 --- [pool-Check host status28,,159ce6638647] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2023-11-27T13:39:14.830+08:00 2023-11-27 13:39:14.836 INFO 10 --- [pool-Check host status20,,0908082dfebe] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-11-27 13:39:14.840 INFO 10 --- [pool-Check host status28,,159ce6638647] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 14 last available time 2023-11-27T13:39:14.830+08:00 and status AVAILABLE 2023-11-27 13:39:14.849 INFO 10 --- [pool-Check host status22,,b18a4b8381b8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-11-27 13:39:14.850 INFO 10 --- [pool-Check host status28,,159ce6638647] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=7 2023-11-27 13:39:14.859 INFO 10 --- [pool-Check host status23,,ae034093a873] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-11-27 13:39:14.868 INFO 10 --- [pool-Check host status28,,159ce6638647] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2023-11-27 13:39:14.881 INFO 10 --- [pool-Check host status20,,0908082dfebe] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=37 2023-11-27 13:39:14.906 INFO 10 --- [pool-Check host status20,,0908082dfebe] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=37 2023-11-27 13:39:14.907 INFO 10 --- [pool-Check host status22,,b18a4b8381b8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=31 2023-11-27 13:39:14.936 INFO 10 --- [pool-Check host status22,,b18a4b8381b8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=31 2023-11-27 13:39:14.944 INFO 10 --- [pool-Check host status28,,159ce6638647] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-11-27 13:39:14.951 INFO 10 --- [pool-Check host status23,,ae034093a873] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-11-27 13:39:14.975 INFO 10 --- [pool-Check host status20,,0908082dfebe] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-11-27 13:39:14.992 INFO 10 --- [pool-Check host status23,,ae034093a873] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=34 2023-11-27 13:39:15.002 INFO 10 --- [pool-Check host status28,,159ce6638647] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2023-11-27 13:39:15.019 INFO 10 --- [pool-Check host status23,,ae034093a873] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=34 2023-11-27 13:39:15.021 INFO 10 --- [pool-Check host status28,,159ce6638647] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2023-11-27 13:39:15.027 INFO 10 --- [pool-Check host status22,,b18a4b8381b8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-11-27 13:39:15.038 INFO 10 --- [pool-Check host status20,,0908082dfebe] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=38 2023-11-27 13:39:15.050 INFO 10 --- [pool-Check host status28,,159ce6638647] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-11-27 13:39:15.053 INFO 10 --- [pool-Check host status23,,ae034093a873] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-11-27 13:39:15.058 INFO 10 --- [pool-Check host status20,,0908082dfebe] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=38 2023-11-27 13:39:15.074 INFO 10 --- [pool-Check host status22,,b18a4b8381b8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=32 2023-11-27 13:39:15.092 INFO 10 --- [pool-Check host status20,,0908082dfebe] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-11-27 13:39:15.097 INFO 10 --- [pool-Check host status28,,159ce6638647] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2023-11-27 13:39:15.098 INFO 10 --- [pool-Check host status23,,ae034093a873] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=35 2023-11-27 13:39:15.109 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063555, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:15.109 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063555, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:15.109 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063555, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:15.109 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063555, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:15.109 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063555, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:15.112 INFO 10 --- [pool-Check host status22,,b18a4b8381b8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=32 2023-11-27 13:39:15.135 INFO 10 --- [pool-Check host status20,,0908082dfebe] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=39 2023-11-27 13:39:15.137 INFO 10 --- [pool-Check host status23,,ae034093a873] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=35 2023-11-27 13:39:15.145 INFO 10 --- [pool-Check host status28,,159ce6638647] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2023-11-27 13:39:15.159 INFO 10 --- [pool-Check host status20,,0908082dfebe] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=39 2023-11-27 13:39:15.165 INFO 10 --- [pool-Check host status22,,b18a4b8381b8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-11-27 13:39:15.191 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:39:15.203 INFO 10 --- [pool-Check host status22,,b18a4b8381b8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=33 2023-11-27 13:39:15.207 INFO 10 --- [pool-Check host status23,,ae034093a873] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-11-27 13:39:15.215 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:39:15.217 INFO 10 --- [pool-Check host status22,,b18a4b8381b8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=33 2023-11-27 13:39:15.217 INFO 10 --- [pool-Check host status28,,159ce6638647] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-11-27 13:39:15.250 INFO 10 --- [pool-Check host status23,,ae034093a873] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=36 2023-11-27 13:39:15.263 INFO 10 --- [pool-Check host status23,,ae034093a873] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=36 2023-11-27 13:39:15.266 INFO 10 --- [pool-Check host status28,,159ce6638647] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2023-11-27 13:39:15.290 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:39:15.297 INFO 10 --- [pool-Check host status28,,159ce6638647] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2023-11-27 13:39:15.316 INFO 10 --- [pool-Check host status26,,cb8a133796c4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=133.197.206.97, originalStatus=OFFLINE 2023-11-27 13:39:15.317 INFO 10 --- [pool-Check host status2,,2d2f843e4353] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=133.197.204.3, originalStatus=OFFLINE 2023-11-27 13:39:15.317 INFO 10 --- [pool-Check host status30,,a708196d88dd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=133.197.206.92, originalStatus=OFFLINE 2023-11-27 13:39:15.318 INFO 10 --- [pool-Check host status27,,91d315ee880c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=133.197.204.10, originalStatus=ONLINE 2023-11-27 13:39:15.319 INFO 10 --- [pool-Check host status25,,bc8f1b698ab5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=133.197.204.11, originalStatus=ONLINE 2023-11-27 13:39:15.319 INFO 10 --- [pool-Check host status3,,fbc25171cb66] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=6, ip=133.197.204.12, originalStatus=ONLINE 2023-11-27 13:39:15.319 INFO 10 --- [pool-Check host status12,,0506ad00d873] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=133.197.204.13, originalStatus=ONLINE 2023-11-27 13:39:15.319 INFO 10 --- [pool-Check host status1,,be0bc6b55f19] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=9, ip=133.197.206.88, originalStatus=OFFLINE 2023-11-27 13:39:15.329 INFO 10 --- [pool-Check host status1,,be0bc6b55f19] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:39:15.335 INFO 10 --- [pool-Check host status12,,0506ad00d873] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2023-11-27 13:39:15.337 INFO 10 --- [pool-Check host status26,,cb8a133796c4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:39:15.343 INFO 10 --- [pool-Check host status3,,fbc25171cb66] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2023-11-27 13:39:15.344 INFO 10 --- [pool-Check host status30,,a708196d88dd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:39:15.344 INFO 10 --- [pool-Check host status2,,2d2f843e4353] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:39:15.348 INFO 10 --- [pool-Check host status25,,bc8f1b698ab5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-11-27 13:39:15.357 INFO 10 --- [pool-Check host status27,,91d315ee880c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-11-27 13:39:15.424 INFO 10 --- [pool-Check host status1,,be0bc6b55f19] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.88:62888/api/v1/agent/status, params:null 2023-11-27 13:39:15.424 INFO 10 --- [pool-Check host status30,,a708196d88dd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.92:62888/api/v1/agent/status, params:null 2023-11-27 13:39:15.427 WARN 10 --- [pool-Check host status1,,be0bc6b55f19] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent status check failed. host=9, ip=133.197.206.88, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.88:62888 [/133.197.206.88] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.agentStatus(AgentExecutor.java:622) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostAgentStatus(HostCheckManagerImpl.java:83) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.88:62888 [/133.197.206.88] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 23 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 32 common frames omitted 2023-11-27 13:39:15.433 WARN 10 --- [pool-Check host status30,,a708196d88dd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent status check failed. host=3, ip=133.197.206.92, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.92:62888 [/133.197.206.92] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.agentStatus(AgentExecutor.java:622) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostAgentStatus(HostCheckManagerImpl.java:83) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.92:62888 [/133.197.206.92] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 23 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 32 common frames omitted 2023-11-27 13:39:15.434 INFO 10 --- [pool-Check host status1,,be0bc6b55f19] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [9:133.197.206.88], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:39:15.434 INFO 10 --- [pool-Check host status1,,be0bc6b55f19] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:39:15.436 INFO 10 --- [pool-Check host status1,,be0bc6b55f19] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=9 2023-11-27 13:39:15.437 INFO 10 --- [pool-Check host status1,,be0bc6b55f19] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=9 2023-11-27 13:39:15.444 INFO 10 --- [pool-Check host status1,,be0bc6b55f19] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:39:15.446 INFO 10 --- [pool-Check host status30,,a708196d88dd] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [3:133.197.206.92], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:39:15.446 INFO 10 --- [pool-Check host status30,,a708196d88dd] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:39:15.453 INFO 10 --- [pool-Check host status30,,a708196d88dd] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=3 2023-11-27 13:39:15.453 INFO 10 --- [pool-Check host status30,,a708196d88dd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2023-11-27 13:39:15.463 INFO 10 --- [pool-Check host status30,,a708196d88dd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:39:15.466 INFO 10 --- [pool-Check host status27,,91d315ee880c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.10:62888/api/v1/agent/status, params:null 2023-11-27 13:39:15.470 INFO 10 --- [pool-Check host status12,,0506ad00d873] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.13:62888/api/v1/agent/status, params:null 2023-11-27 13:39:15.472 INFO 10 --- [pool-Check host status27,,91d315ee880c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=133.197.204.10 2023-11-27 13:39:15.472 INFO 10 --- [pool-Check host status27,,91d315ee880c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=133.197.204.10 2023-11-27 13:39:15.472 INFO 10 --- [pool-Check host status27,,91d315ee880c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=133.197.204.10 2023-11-27 13:39:15.475 INFO 10 --- [pool-Check host status2,,2d2f843e4353] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.3:62888/api/v1/agent/status, params:null 2023-11-27 13:39:15.479 WARN 10 --- [pool-Check host status2,,2d2f843e4353] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent status check failed. host=2, ip=133.197.204.3, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.204.3:62888 [/133.197.204.3] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.agentStatus(AgentExecutor.java:622) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostAgentStatus(HostCheckManagerImpl.java:83) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.204.3:62888 [/133.197.204.3] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 23 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 32 common frames omitted 2023-11-27 13:39:15.481 INFO 10 --- [pool-Check host status12,,0506ad00d873] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=133.197.204.13 2023-11-27 13:39:15.481 INFO 10 --- [pool-Check host status12,,0506ad00d873] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=133.197.204.13 2023-11-27 13:39:15.482 INFO 10 --- [pool-Check host status12,,0506ad00d873] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=133.197.204.13 2023-11-27 13:39:15.486 INFO 10 --- [pool-Check host status25,,bc8f1b698ab5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.11:62888/api/v1/agent/status, params:null 2023-11-27 13:39:15.490 INFO 10 --- [pool-Check host status26,,cb8a133796c4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.97:62888/api/v1/agent/status, params:null 2023-11-27 13:39:15.492 WARN 10 --- [pool-Check host status26,,cb8a133796c4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent status check failed. host=1, ip=133.197.206.97, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.97:62888 [/133.197.206.97] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.agentStatus(AgentExecutor.java:622) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostAgentStatus(HostCheckManagerImpl.java:83) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.97:62888 [/133.197.206.97] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 23 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 32 common frames omitted 2023-11-27 13:39:15.493 INFO 10 --- [pool-Check host status25,,bc8f1b698ab5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=133.197.204.11 2023-11-27 13:39:15.493 INFO 10 --- [pool-Check host status25,,bc8f1b698ab5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=133.197.204.11 2023-11-27 13:39:15.493 INFO 10 --- [pool-Check host status25,,bc8f1b698ab5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=133.197.204.11 2023-11-27 13:39:15.494 INFO 10 --- [pool-Check host status2,,2d2f843e4353] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [2:133.197.204.3], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:39:15.494 INFO 10 --- [pool-Check host status2,,2d2f843e4353] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:39:15.507 INFO 10 --- [pool-Check host status26,,cb8a133796c4] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [1:133.197.206.97], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:39:15.507 INFO 10 --- [pool-Check host status26,,cb8a133796c4] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:39:15.508 INFO 10 --- [pool-Check host status3,,fbc25171cb66] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.12:62888/api/v1/agent/status, params:null 2023-11-27 13:39:15.518 INFO 10 --- [pool-Check host status3,,fbc25171cb66] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=133.197.204.12 2023-11-27 13:39:15.518 INFO 10 --- [pool-Check host status3,,fbc25171cb66] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=133.197.204.12 2023-11-27 13:39:15.518 INFO 10 --- [pool-Check host status3,,fbc25171cb66] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=133.197.204.12 2023-11-27 13:39:15.521 INFO 10 --- [pool-Check host status26,,cb8a133796c4] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=1 2023-11-27 13:39:15.521 INFO 10 --- [pool-Check host status26,,cb8a133796c4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2023-11-27 13:39:15.524 INFO 10 --- [pool-Check host status2,,2d2f843e4353] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=2 2023-11-27 13:39:15.524 INFO 10 --- [pool-Check host status2,,2d2f843e4353] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2023-11-27 13:39:15.525 INFO 10 --- [pool-Check host status26,,cb8a133796c4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:39:15.533 INFO 10 --- [pool-Check host status2,,2d2f843e4353] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:39:15.743 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:39:15.951 INFO 10 --- [pool-ocp-schedules-16,,0094ba1597fd] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:39:15.951 INFO 10 --- [pool-ocp-schedules-16,,0094ba1597fd] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:39:16.126 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:39:16.127 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:39:16.334 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:39:19.127 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:39:19.130 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:39:19.337 INFO 10 --- [http-nio-0.0.0.0-8080-exec-6,2ecdade7680d4a3f,aa8692ab215c] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=133.197.206.97, traceId=2ecdade7680d4a3f, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-11-27 13:39:19.344 INFO 10 --- [http-nio-0.0.0.0-8080-exec-6,2ecdade7680d4a3f,aa8692ab215c] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.206.97, traceId=2ecdade7680d4a3f, duration=8 ms] 2023-11-27 13:39:20.011 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:38:20.0, nextRun=2023-11-27 13:39:20.0, currentDate=Mon Nov 27 13:39:20 CST 2023 2023-11-27 13:39:20.012 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-11-27 13:38:20.0, nextRunTime=2023-11-27 13:39:20.0} 2023-11-27 13:39:20.064 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-11-27 13:39:20.064 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:39:20.064 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-11-27 13:39:20.103 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063560, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:20.103 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063560, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:20.104 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063560, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:20.110 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063560, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:20.110 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063560, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:20.913 INFO 10 --- [http-nio-0.0.0.0-8080-exec-1,b4c7b740309c43ac,613aacdf5ae5] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.13, traceId=b4c7b740309c43ac, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:39:20.934 INFO 10 --- [http-nio-0.0.0.0-8080-exec-1,b4c7b740309c43ac,613aacdf5ae5] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.13, traceId=b4c7b740309c43ac, duration=21 ms] 2023-11-27 13:39:21.124 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000067, operation:EXECUTE, state: RUNNING 2023-11-27 13:39:21.126 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000067, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-11-27T13:39:21.121+08:00, endTime=null} 2023-11-27 13:39:21.179 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,ae74b55f25be] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000067, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000057, task_operation=execute, latest_execution_start_time=2023-11-27T13:39:21.120+08:00, sub_task_instance_id=10000067}, listMap={}}, executor=10.120.0.1 2023-11-27 13:39:21.181 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,ae74b55f25be] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-11-27 13:39:21.181 INFO 10 --- [ForkJoinPool-35-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-11-27 13:39:21.186 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,ae74b55f25be] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000067, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:39:21.254 INFO 10 --- [ForkJoinPool-35-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:21.254 INFO 10 --- [ForkJoinPool-35-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:21.254 INFO 10 --- [ForkJoinPool-35-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:21.255 INFO 10 --- [ForkJoinPool-35-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:21.261 INFO 10 --- [ForkJoinPool-35-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:21.262 INFO 10 --- [ForkJoinPool-35-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:21.262 INFO 10 --- [ForkJoinPool-35-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT MAX(CASE `name` WHEN 'cluster' THEN `info` ELSE '' END ) `cluster`, MAX(CASE `name` WHEN 'merge_status' THEN `info` ELSE '' END ) `merge_status`, MAX(CASE `name` WHEN 'global_broadcast_version' THEN `value` ELSE 0 END ) `global_broadcast_version`, MAX(CASE `name` WHEN 'frozen_version' THEN `value` ELSE 0 END ) `frozen_version`, MAX(CASE `name` WHEN 'last_merged_version' THEN `value` ELSE 0 END ) `last_merged_version`, MAX(CASE `name` WHEN 'frozen_time' THEN `value` ELSE 0 END ) `frozen_time`, MAX(CASE `name` WHEN 'is_merge_error' THEN `value` ELSE 0 END ) `merge_error` FROM oceanbase.__all_zone WHERE `zone` = '' 2023-11-27 13:39:21.263 INFO 10 --- [ForkJoinPool-35-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:21.263 INFO 10 --- [ForkJoinPool-35-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:21.264 INFO 10 --- [ForkJoinPool-35-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select column_value from oceanbase.__all_core_table where table_name ='__all_global_stat' and column_name = 'frozen_version' 2023-11-27 13:39:21.268 INFO 10 --- [ForkJoinPool-35-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:21.268 INFO 10 --- [ForkJoinPool-35-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:21.269 INFO 10 --- [ForkJoinPool-35-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, MAX(CASE `name` WHEN 'region' THEN `info` ELSE '' END ) `region`, MAX(CASE `name` WHEN 'idc' THEN `info` ELSE '' END ) `idc`, MAX(CASE `name` WHEN 'status' THEN `info` ELSE '' END ) `status`, MAX(CASE `name` WHEN 'merge_status' THEN `info` ELSE '' END ) `merge_status`, MAX(CASE `name` WHEN 'broadcast_version' THEN `value` ELSE 0 END ) `broadcast_version`, MAX(CASE `name` WHEN 'all_merged_version' THEN `value` ELSE 0 END ) `all_merged_version`, MAX(CASE `name` WHEN 'last_merged_version' THEN `value` ELSE 0 END ) `last_merged_version`, MAX(CASE `name` WHEN 'merge_start_time' THEN `value` ELSE 0 END ) `merge_start_time`, MAX(CASE `name` WHEN 'last_merged_time' THEN `value` ELSE 0 END ) `last_merged_time`, MAX(CASE `name` WHEN 'is_merge_timeout' THEN `value` ELSE 0 END ) `merge_timeout` FROM oceanbase.__all_zone WHERE `zone` <> '' GROUP BY `zone` 2023-11-27 13:39:21.270 INFO 10 --- [ForkJoinPool-35-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : compaction collect success, cluster=ObClusterOverview(id=1, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"133.197.206.97:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1700793672036000}, createTime=2022-11-29T14:52:53+08:00, creator=admin, obVersion=3.1.4, tenantCount=3) 2023-11-27 13:39:21.724 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000057, status:SUCCESS 2023-11-27 13:39:21.726 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000057, status:SUCCESS 2023-11-27 13:39:21.797 INFO 10 --- [pool-overview-container-6,,a60e24d828dc] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:39:21.841 INFO 10 --- [pool-scanner--12040-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=88 2023-11-27 13:39:22.130 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:39:22.131 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:39:22.778 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:39:22.777447+08:00, lastProcessTime=2023-11-27T13:39:12.777842+08:00 2023-11-27 13:39:22.879 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:39:22.777447+08:00 2023-11-27 13:39:23.132 INFO 10 --- [http-nio-0.0.0.0-8080-exec-3,19bd2f43a02f4bb9,6a1c72e66c49] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.11, traceId=19bd2f43a02f4bb9, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:39:23.137 INFO 10 --- [http-nio-0.0.0.0-8080-exec-3,19bd2f43a02f4bb9,6a1c72e66c49] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.11, traceId=19bd2f43a02f4bb9, duration=6 ms] 2023-11-27 13:39:23.528 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.120.0.1:8080 2023-11-27 13:39:23.674 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.120.0.1:8080 2023-11-27 13:39:23.915 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.120.0.1:8080 2023-11-27 13:39:24.236 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:39:24.652 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.120.0.1:8080 2023-11-27 13:39:24.656 INFO 10 --- [pool-ocp-schedules-20,,0abc298eb50a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:39:24.671 INFO 10 --- [alarm-detect-12,,1d85d8273a50] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_cluster_exists_inactive_server detect time range 1701063544 ~ 1701063554 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={inactive_server_count > 0=[ExpressionEvaluateResult(matchValue=0.0, metric=inactive_server_count, match=true, value=1.0, values={1701063550=1.0}, valueCount=1, labels={app=OB, ob_cluster_id=10, ob_cluster_name=darenOB, server_ips=133.197.204.3})]}, matched=true, message=null, level=2) 2023-11-27 13:39:25.104 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063565, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:25.104 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063565, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:25.105 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063565, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:25.105 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063565, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:25.113 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063565, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:25.131 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:39:25.133 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:39:25.216 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:39:25.220 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:39:25.321 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:39:25.439 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:38:26.0, nextRun=2023-11-27 13:39:25.0, currentDate=Mon Nov 27 13:39:25 CST 2023 2023-11-27 13:39:25.439 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-11-27 13:38:26.0, nextRunTime=2023-11-27 13:39:25.0} 2023-11-27 13:39:25.500 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-11-27 13:39:25.500 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:39:25.500 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-11-27 13:39:25.751 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:39:26.215 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000068, operation:EXECUTE, state: RUNNING 2023-11-27 13:39:26.217 INFO 10 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000068, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-11-27T13:39:26.207+08:00, endTime=null} 2023-11-27 13:39:26.263 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,3d186baf31bb] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000068, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000058, task_operation=execute, latest_execution_start_time=2023-11-27T13:39:26.202+08:00, sub_task_instance_id=10000068}, listMap={host_ids=[1, 2, 3, 4, 5, 6, 7, 9]}}, executor=10.120.0.1 2023-11-27 13:39:26.338 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,3d186baf31bb] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270991, maxInactiveSeconds=180 2023-11-27 13:39:26.338 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,3d186baf31bb] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270991, maxInactiveSeconds=180 2023-11-27 13:39:26.338 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,3d186baf31bb] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270991, maxInactiveSeconds=180 2023-11-27 13:39:26.338 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:39:26.338 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,3d186baf31bb] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270991, maxInactiveSeconds=180 2023-11-27 13:39:26.338 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,3d186baf31bb] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270991, maxInactiveSeconds=180 2023-11-27 13:39:26.338 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,3d186baf31bb] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270991, maxInactiveSeconds=180 2023-11-27 13:39:26.338 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,3d186baf31bb] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270991, maxInactiveSeconds=180 2023-11-27 13:39:26.338 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,3d186baf31bb] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=270991, maxInactiveSeconds=180 2023-11-27 13:39:26.338 WARN 10 --- [pool-subtask-executor15,554481c3fd1b426b,3d186baf31bb] c.o.o.s.t.b.host.CheckExporterTask : Host [id=1, ip=133.197.206.97, status=OFFLINE] not online, skip check exporter 2023-11-27 13:39:26.339 WARN 10 --- [pool-subtask-executor15,554481c3fd1b426b,3d186baf31bb] c.o.o.s.t.b.host.CheckExporterTask : Host [id=2, ip=133.197.204.3, status=OFFLINE] not online, skip check exporter 2023-11-27 13:39:26.339 WARN 10 --- [pool-subtask-executor15,554481c3fd1b426b,3d186baf31bb] c.o.o.s.t.b.host.CheckExporterTask : Host [id=3, ip=133.197.206.92, status=OFFLINE] not online, skip check exporter 2023-11-27 13:39:26.340 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,3d186baf31bb] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=133.197.204.10] desired exporter count: 8, actual exporter count: 8 2023-11-27 13:39:26.342 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,3d186baf31bb] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=133.197.204.11] desired exporter count: 8, actual exporter count: 8 2023-11-27 13:39:26.342 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,3d186baf31bb] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=6, ip=133.197.204.12] desired exporter count: 8, actual exporter count: 8 2023-11-27 13:39:26.343 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,3d186baf31bb] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=7, ip=133.197.204.13] desired exporter count: 8, actual exporter count: 8 2023-11-27 13:39:26.343 WARN 10 --- [pool-subtask-executor15,554481c3fd1b426b,3d186baf31bb] c.o.o.s.t.b.host.CheckExporterTask : Host [id=9, ip=133.197.206.88, status=OFFLINE] not online, skip check exporter 2023-11-27 13:39:26.354 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,3d186baf31bb] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000068, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:39:26.915 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000058, status:SUCCESS 2023-11-27 13:39:26.916 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000058, status:SUCCESS 2023-11-27 13:39:28.133 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:39:28.144 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:39:30.102 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063570, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:30.102 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063570, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:30.103 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063570, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:30.105 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063570, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:30.105 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063570, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:30.737 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:38:30.0, nextRun=2023-11-27 13:39:30.0, currentDate=Mon Nov 27 13:39:30 CST 2023 2023-11-27 13:39:30.737 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-11-27 13:38:30.0, nextRunTime=2023-11-27 13:39:30.0} 2023-11-27 13:39:30.768 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-11-27 13:39:30.768 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:39:30.768 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:39:30.768 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-11-27 13:39:30.796 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:39:01.0, nextRun=2023-11-27 13:39:30.0, currentDate=Mon Nov 27 13:39:30 CST 2023 2023-11-27 13:39:30.796 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-11-27 13:39:01.0, nextRunTime=2023-11-27 13:39:30.0} 2023-11-27 13:39:30.801 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-11-27 13:39:30.801 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:39:30.801 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-11-27 13:39:31.145 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:39:31.147 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:39:31.306 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000069, operation:EXECUTE, state: RUNNING 2023-11-27 13:39:31.308 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000069, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-11-27T13:39:31.304+08:00, endTime=null} 2023-11-27 13:39:31.324 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000070, operation:EXECUTE, state: RUNNING 2023-11-27 13:39:31.327 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000070, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-11-27T13:39:31.322+08:00, endTime=null} 2023-11-27 13:39:31.342 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000069, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=8000059, task_operation=execute, latest_execution_start_time=2023-11-27T13:39:31.303+08:00, sub_task_instance_id=10000069}, listMap={cluster_ids=[1, 2]}}, executor=10.120.0.1 2023-11-27 13:39:31.355 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000071, operation:EXECUTE, state: RUNNING 2023-11-27 13:39:31.360 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000071, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-11-27T13:39:31.347+08:00, endTime=null} 2023-11-27 13:39:31.377 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000070, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=8000059, task_operation=execute, latest_execution_start_time=2023-11-27T13:39:31.322+08:00, sub_task_instance_id=10000070}, listMap={cluster_ids=[1, 2]}}, executor=10.120.0.1 2023-11-27 13:39:31.391 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b1ce8bd08a65] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000071, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000060, task_operation=execute, latest_execution_start_time=2023-11-27T13:39:31.345+08:00, sub_task_instance_id=10000071}, listMap={}}, executor=10.120.0.1 2023-11-27 13:39:31.391 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b1ce8bd08a65] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-11-27 13:39:31.423 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.424 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.424 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.424 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:31.430 INFO 10 --- [pool-1085-thread-1,554481c3fd1b426b,86c285549cfa] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:darenOB, tenant:sys 2023-11-27 13:39:31.430 INFO 10 --- [pool-1085-thread-2,554481c3fd1b426b,45ea1e74ffc6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:darenOB, tenant:daren 2023-11-27 13:39:31.439 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.440 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.446 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, tenant_name, (CASE WHEN tenant_id = 1 THEN 'SYS' ELSE 'USER' END) AS tenant_type, (CASE WHEN compatibility_mode = 1 THEN 'ORACLE' ELSE 'MYSQL' END) AS compatibility_mode, zone_list, primary_zone, locality, status, locked, read_only, in_recyclebin, TIMESTAMPDIFF(SECOND, gmt_create, now()) AS exist_seconds FROM oceanbase.__all_tenant 2023-11-27 13:39:31.455 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.455 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.456 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1] 2023-11-27 13:39:31.465 INFO 10 --- [pool-1085-thread-1,554481c3fd1b426b,86c285549cfa] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:31.466 INFO 10 --- [pool-1085-thread-1,554481c3fd1b426b,86c285549cfa] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:31.466 INFO 10 --- [pool-1085-thread-1,554481c3fd1b426b,86c285549cfa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.466 INFO 10 --- [pool-1085-thread-1,554481c3fd1b426b,86c285549cfa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:31.474 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.475 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.476 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1] 2023-11-27 13:39:31.477 INFO 10 --- [pool-1085-thread-1,554481c3fd1b426b,86c285549cfa] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:31.477 INFO 10 --- [pool-1085-thread-1,554481c3fd1b426b,86c285549cfa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.481 INFO 10 --- [pool-1085-thread-1,554481c3fd1b426b,86c285549cfa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-11-27 13:39:31.487 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.488 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.488 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1001] 2023-11-27 13:39:31.490 INFO 10 --- [pool-1085-thread-2,554481c3fd1b426b,45ea1e74ffc6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:31.491 INFO 10 --- [pool-1085-thread-2,554481c3fd1b426b,45ea1e74ffc6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:31.491 INFO 10 --- [pool-1085-thread-2,554481c3fd1b426b,45ea1e74ffc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.491 INFO 10 --- [pool-1085-thread-2,554481c3fd1b426b,45ea1e74ffc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:31.502 INFO 10 --- [pool-1085-thread-1,554481c3fd1b426b,86c285549cfa] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1701021601748952808, broadcast scn:1701021601748952808, saved frozen scn:1701021601748952808 2023-11-27 13:39:31.507 INFO 10 --- [pool-1085-thread-1,554481c3fd1b426b,86c285549cfa] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:darenOB, tenant:sys 2023-11-27 13:39:31.509 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.509 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.509 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1001] 2023-11-27 13:39:31.513 INFO 10 --- [pool-1085-thread-2,554481c3fd1b426b,45ea1e74ffc6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:31.513 INFO 10 --- [pool-1085-thread-2,554481c3fd1b426b,45ea1e74ffc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.520 INFO 10 --- [pool-1085-thread-2,554481c3fd1b426b,45ea1e74ffc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-11-27 13:39:31.521 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.521 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.522 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1002] 2023-11-27 13:39:31.535 INFO 10 --- [pool-1085-thread-2,554481c3fd1b426b,45ea1e74ffc6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1701021600396785720, broadcast scn:1701021600396785720, saved frozen scn:1701021600396785720 2023-11-27 13:39:31.535 INFO 10 --- [pool-1085-thread-2,554481c3fd1b426b,45ea1e74ffc6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:darenOB, tenant:daren 2023-11-27 13:39:31.539 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.539 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.546 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1002] 2023-11-27 13:39:31.552 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b1ce8bd08a65] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-11-27 13:39:31.562 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-11-27 13:39:31.565 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:31.568 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:31.568 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.575 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:31.582 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:39:31.583 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.583 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.583 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.584 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:31.589 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.589 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.590 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-11-27 13:39:31.590 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:31.590 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.591 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b1ce8bd08a65] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000071, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:39:31.591 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-11-27 13:39:31.602 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:39:31.603 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.603 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.603 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.603 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:31.605 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.605 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.605 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:39:31.605 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:31.605 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.610 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-11-27 13:39:31.624 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:39:31.625 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.625 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.625 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.625 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:31.627 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.627 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.627 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1001, ALTER_TENANT_LOCALITY] 2023-11-27 13:39:31.627 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:31.627 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.628 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-11-27 13:39:31.639 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:39:31.640 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.640 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.640 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.640 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:31.642 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.642 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.642 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1001, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:39:31.648 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:31.648 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.659 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-11-27 13:39:31.664 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:39:31.664 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.665 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.665 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.665 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:31.666 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.666 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.667 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-11-27 13:39:31.679 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:39:31.680 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.680 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.680 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.681 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:31.682 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.682 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.683 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:39:31.683 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-11-27 13:39:31.685 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:31.685 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.691 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-11-27 13:39:31.702 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-11-27 13:39:31.740 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.120.0.1 2023-11-27 13:39:31.757 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:39:31.758 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:31.759 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:31.759 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.760 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:31.769 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:39:31.770 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.771 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.771 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.771 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:31.774 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.774 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.775 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1] 2023-11-27 13:39:31.785 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:39:31.788 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:31.788 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.789 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-11-27 13:39:31.793 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-11-27 13:39:31.794 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.794 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.795 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1001] 2023-11-27 13:39:31.798 INFO 10 --- [pool-overview-container-5,,c2c7d57e2a1b] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:39:31.809 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:39:31.814 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-11-27 13:39:31.815 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:31.815 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.816 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1002] 2023-11-27 13:39:31.827 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:39:31.832 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-11-27 13:39:31.833 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-11-27 13:39:31.833 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=obcluster, obClusterId=1 2023-11-27 13:39:31.842 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,e25a4d93c7bf] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000069, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:39:31.851 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:39:31.852 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:31.852 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:31.852 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.853 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:31.857 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:31.857 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.858 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:39:31.874 INFO 10 --- [pool-scanner--12133-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=88 2023-11-27 13:39:31.918 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:39:31.919 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:31.920 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:31.920 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.920 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:31.924 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:31.924 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.925 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-11-27 13:39:31.961 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:39:31.962 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:31.962 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:31.963 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.963 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:31.966 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:31.967 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:31.967 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:39:31.968 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-11-27 13:39:31.972 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.120.0.1 2023-11-27 13:39:32.005 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:39:32.006 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:32.009 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:32.009 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:32.010 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:32.016 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:32.016 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:32.017 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-11-27 13:39:32.026 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:39:32.030 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-11-27 13:39:32.031 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:32.031 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:32.032 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-11-27 13:39:32.043 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:39:32.046 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-11-27 13:39:32.046 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-11-27 13:39:32.046 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=darenOB, obClusterId=10 2023-11-27 13:39:32.050 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,6ec7376659a8] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000070, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:39:32.133 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000059, status:SUCCESS 2023-11-27 13:39:32.135 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000059, status:SUCCESS 2023-11-27 13:39:32.147 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000060, status:SUCCESS 2023-11-27 13:39:32.148 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000060, status:SUCCESS 2023-11-27 13:39:32.799 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:39:32.797688+08:00, lastProcessTime=2023-11-27T13:39:22.777447+08:00 2023-11-27 13:39:32.959 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:39:32.797688+08:00 2023-11-27 13:39:33.974 INFO 10 --- [pool-prometheus-cache-monitor1,,f0049f873bca] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":5121,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":1707,"seriesIdKeySize":87133,"secondWriteQueue":0,"minuteWriteCacheSize":32707} 2023-11-27 13:39:33.974 INFO 10 --- [pool-prometheus-cache-monitor1,,f0049f873bca] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_zone_current_timestamp=38, ob_server_resource_memory_assigned_bytes=21, tcp_retrans=30, ob_unit_num=21, go_memstats_mspan_sys_bytes=17, go_memstats_stack_sys_bytes=17, exporter_export_metrics_bytes_total=28, node_filesystem_size_bytes=87, http_server_requests_seconds_count=40, ob_compaction_frozen_time=18, ob_system_event_total_waits=34, monagent_input_sql_duration_seconds_sum=60, observer_fd_count=20, ob_plan_cache_hit_total=28, processor_process_metrics_total=22, node_filesystem_readonly=86, ob_server_resource_disk_in_use_bytes=19, go_memstats_frees_total=17, sql_audit_input_collect_data_jumped_total=12, io_await=60, host_available=13, go_memstats_mallocs_total=17, mysql_output_write_sql_failed_count=23, ob_unit_config_max_memory_bytes=45, io_util=60, ob_system_event_detail_total_timeouts=782, ob_tenant_disk_v4_data_size=18, ob_latch_sleep_total=326, ob_tenant_unit_min_memory=41, ob_memstore_total_bytes=28, process_resident_memory_bytes=17, ob_cache_size_bytes=47, ob_tenant_unit_max_memory=41, ob_tenant_assigned_cpu_total=25, input_collect_metrics_total=29, ob_tenant_server_available_disk_free=28, ob_tenant_server_avg_cpu_used_percentage=23, node_disk_io_time_weighted_seconds_total=39, node_disk_write_time_seconds_total=39, cpu_count=30, output_write_metrics_total=15, ob_cluster_sync=175, process_open_fds=17, odp_transaction_total=22, ob_system_event_detail_time_waited_us=782, process_max_fds=17, ob_server_resource_cpu_assigned_percent=20, ocp_alarm_detect_duration_ms_count=82, go_memstats_heap_inuse_bytes=17, ob_tenant_server_max_cpu_used_percentage=23, proxy_memory_used_bytes=30, go_memstats_mcache_sys_bytes=17, ob_tenant_disk_v4_log_size=18, node_memory_memfree_bytes=29, process_exists=38, go_gc_duration_seconds=22, ob_server_resource_disk_bytes=21, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=27, node_memory_cached_bytes=29, ob_table_num=24, ob_disk_free_bytes=21, node_filesystem_avail_bytes=87, ob_backup_storage_capacity_bytes=349, ob_active_session_num=27, ob_unit_config_max_iops=45, ob_plan_cache_memory_bytes=28, node_network_transmit_errs_total=65, go_memstats_last_gc_time_seconds=17, go_gc_duration_seconds_count=17, monitor_exporter_avaliable=78, ob_tenant_server_max_memory_used_percentage=23, node_disk_read_bytes_total=39, mysql_output_write_sql_milliseconds_summary_count=25, ob_compaction_is_suspended=18, ob_tenant_unit_log_disk_size=22, go_memstats_lookups_total=17, mysql_output_write_sql_count=25, log_tailer_count=17, host_agent_process_status=21, ob_server_resource_cpu=21, go_memstats_alloc_bytes=17, ob_server_num=58, node_cpu_seconds_total_native=47, ob_tenant_unit_count=39, ob_session_all_num=25, ob_tenant_resource_min_memory=25, ob_memstore_freeze_times=28, ob_system_event_time_waited=34, ob_tenant_resource_min_cpu=25, ob_cluster_status_check=49, monagent_plugin_execute_seconds_total=41, node_net_bandwidth_bps=45, monagent_oceanbase_db_pool_idle=7, go_memstats_heap_idle_bytes=17, go_memstats_gc_sys_bytes=17, ob_server_resource_memory_bytes=21, ob_tenant_sys_partition_required_size=12, ob_latch_get_total=326, go_info=18, ob_tenant_assigned_mem_assigned=25, log_tailer_processed_file_count=24, node_load15=29, ob_tenant_server_required_size=35, ob_unit_config_min_iops=45, ob_server_partition_replica_limit=7, ob_tenant_unit_max_cpu=41, ob_latch_wait_seconds_total=326, node_memory_buffers_bytes=29, go_memstats_heap_released_bytes=17, node_network_receive_bytes_total=65, ob_memstore_freeze_trigger_bytes=28, ob_waitevent_wait_seconds_total=28, ob_server_stopped_duration_seconds=20, exposer_request_latencies_count=8, ob_role_is_rootserver=20, go_memstats_next_gc_bytes=17, mysql_output_context_timeout_discard_metrics_count=17, ob_session_active_num=25, node_filesystem_files_free=109, ob_memtable_max_snapshot_duration_seconds=27, node_disk_read_time_seconds_total=39, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=22, process_virtual_memory_bytes=17, mysql_output_write_sql_retry_count=23, ob_tenant_assigned_cpu_assigned=25, ob_tenant_unit_min_cpu=41, ob_unit_config_max_cpu=45, ob_unit_config_max_session_num=18, http_request_milliseconds_summary_count=79, monagent_oceanbase_audit_timeinfo_cache_size=7, net_connectivity=10, monagent_input_sql_duration_seconds_bucket=70, ob_latch_immediate_get_total=326, node_cpu_seconds_total=37, http_request_milliseconds_summary=82, ob_latch_immediate_miss_total=326, ob_index_error_num=16, ob_tenant_server_available_disk_total=28, ob_latch_spin_get_total=326, ob_tenant_memory_hold_bytes=18, go_memstats_buck_hash_sys_bytes=17, monagent_pipeline_report_metrics_total=18, mysql_output_write_sql_milliseconds_summary_sum=25, ob_tenant_disk_total_size=25, exposer_transferred_bytes_total=8, odp_sql_request_total=67, mysql_output_write_task_size=24, ob_tenant_context_memory_count=339, node_disk_reads_completed_total=39, exporter_export_metrics_total=28, ob_sysstat=129, ob_partition_num=13, node_ntp_offset_seconds=29, ob_latch_miss_total=326, go_memstats_heap_sys_bytes=17, ocp_meta_db_health=4, ob_disk_total_bytes=21, ob_zone_stat=38, go_memstats_heap_objects=17, go_memstats_mcache_inuse_bytes=17, go_memstats_other_sys_bytes=17, go_threads=17, http_request_milliseconds_summary_sum=79, ob_memstore_active_bytes=28, http_server_requests_seconds_sum=40, go_memstats_alloc_bytes_total=17, ob_tenant_server_partition_replica_count=35, ob_tenant_assigned_mem_total=25, node_memory_memtotal_bytes=29, node_network_transmit_drop_total=65, ob_system_event_detail_total_waits=782, odp_used_connections=28, ob_tenant_task_max_duration_seconds=25, backup_storage_capacity_used_percent=14, exposer_request_latencies=11, ob_waitevent_wait_total=28, odp_sql_cost_total=66, ob_tenant500_memory_used_bytes=20, node_network_transmit_bytes_total=65, ob_compaction_is_error=18, node_network_receive_drop_total=65, ob_server_partition_replica_count=7, monagent_pipeline_execute_total=27, node_filesystem_files=109, ob_tenant_server_avg_memory_used_percentage=23, process_start_time_seconds=17, go_memstats_heap_alloc_bytes=17, ob_tenant_resource_max_cpu=25, go_gc_duration_seconds_sum=17, go_memstats_mspan_inuse_bytes=17, mysql_output_write_sql_milliseconds_summary=28, ob_server_resource_memory_assigned_percent=20, monagent_pipeline_execute_seconds_total=27, go_memstats_sys_bytes=17, node_network_receive_errs_total=65, ob_all_session_num=27, observer_ulimit_max_fd_count=20, ob_plan_cache_access_total=28, ob_tenant_server_row_count=35, ob_tenant_memory_used_bytes=18, monagent_oceanbase_plan_cache_size=7, ob_tenant_server_data_size=35, go_memstats_gc_cpu_fraction=17, monagent_oceanbase_audit_cache_size=7, ob_tenant_context_memory_used_bytes=339, node_disk_writes_completed_total=39, node_load1=29, process_cpu_seconds_total=17, ob_tenant_resource_max_memory=25, go_memstats_stack_inuse_bytes=17, ob_tenant_user_partition_required_size=12, node_load5=29, monagent_input_sql_duration_seconds_count=60, node_disk_written_bytes_total=39, monagent_oceanbase_ash_cache_size=7, ob_tenant_context_memory_hold_bytes=339, go_goroutines=17, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=21, exposer_request_latencies_sum=8, ob_unit_config_min_cpu=45, process_virtual_memory_max_bytes=17, monagent_oceanbase_plan_timeinfo_cache_size=7, monagent_plugin_execute_total=41, ob_tenant500_memory_hold_bytes=20, exposer_scrapes_total=8, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=22, ob_compaction_start_time=18, oceanbase_connectivity=21} 2023-11-27 13:39:34.148 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:39:34.153 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:39:34.235 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:39:34.453 INFO 10 --- [alarm-detect-2,,ea144d182ed4] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_cluster_exists_inactive_server detect time range 1701063554 ~ 1701063564 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={inactive_server_count > 0=[ExpressionEvaluateResult(matchValue=0.0, metric=inactive_server_count, match=true, value=1.0, values={1701063560=1.0}, valueCount=1, labels={app=OB, ob_cluster_id=10, ob_cluster_name=darenOB, server_ips=133.197.204.3})]}, matched=true, message=null, level=2) 2023-11-27 13:39:34.689 INFO 10 --- [pool-ocp-schedules-4,,caa6e478a869] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:39:35.102 INFO 10 --- [pool-prometheus-second-schedule-2,,5ca814b09684] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=11, collectTaskCount=11 2023-11-27 13:39:35.103 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063575, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:35.103 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063575, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:35.103 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063575, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:35.103 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063575, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:35.104 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063575, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:35.110 INFO 10 --- [pool-prometheus-second-schedule-1,,2ed84b6b114b] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=14, collectTaskCount=14 2023-11-27 13:39:35.220 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:39:35.250 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:39:35.344 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:39:35.754 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:39:35.992 INFO 10 --- [pool-ocp-schedules-14,,73c5c5b640e6] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:39:35.993 INFO 10 --- [pool-ocp-schedules-14,,73c5c5b640e6] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:39:36.343 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:39:37.153 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:39:37.154 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:39:39.679 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.120.0.1:8080 2023-11-27 13:39:40.106 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063580, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:40.107 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063580, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:40.111 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063580, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:40.111 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063580, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:40.112 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063580, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:40.154 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:39:40.156 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:39:41.799 INFO 10 --- [pool-overview-container-8,,f8696acace7e] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:39:41.849 INFO 10 --- [pool-scanner--12226-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=88 2023-11-27 13:39:42.821 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:39:42.821243+08:00, lastProcessTime=2023-11-27T13:39:32.797688+08:00 2023-11-27 13:39:42.924 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:39:42.821243+08:00 2023-11-27 13:39:43.160 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:39:43.165 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:39:43.194 INFO 10 --- [http-nio-0.0.0.0-8080-exec-5,7e2ec5c5ab314fcd,968d08fd5624] c.o.o.server.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=obcluster&version=2&ObClusterId=1, client=133.197.206.97, traceId=7e2ec5c5ab314fcd, method=OcpApiRtnObj com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=obcluster,1,2,{"ObRegion":"obcluster","ObCluster":"obcluster","ObRegionId":1,"ObClusterId":1,"Type":"PRIMARY","timestamp":1701063583072660,"RsList":[{"address":"133.197.206.97:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2023-11-27 13:39:43.217 INFO 10 --- [http-nio-0.0.0.0-8080-exec-5,7e2ec5c5ab314fcd,968d08fd5624] c.o.o.obops.configurl.ConfigUrlService : updateRootServer, clusterName=obcluster, clusterId=1, affectRows=1 2023-11-27 13:39:43.220 INFO 10 --- [http-nio-0.0.0.0-8080-exec-5,7e2ec5c5ab314fcd,968d08fd5624] c.o.o.server.trace.RequestTracingAspect : API OK: [POST /services client=133.197.206.97, traceId=7e2ec5c5ab314fcd, duration=32 ms] 2023-11-27 13:39:44.235 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:39:44.650 INFO 10 --- [http-nio-0.0.0.0-8080-exec-7,3982613b57994134,d2ca377b2ed7] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.11, traceId=3982613b57994134, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:39:44.681 INFO 10 --- [http-nio-0.0.0.0-8080-exec-7,3982613b57994134,d2ca377b2ed7] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.11, traceId=3982613b57994134, duration=32 ms] 2023-11-27 13:39:44.718 INFO 10 --- [pool-ocp-schedules-9,,43937a389ab5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:39:44.760 INFO 10 --- [alarm-detect-5,,808992632392] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_cluster_exists_inactive_server detect time range 1701063564 ~ 1701063574 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={inactive_server_count > 0=[ExpressionEvaluateResult(matchValue=0.0, metric=inactive_server_count, match=true, value=1.0, values={1701063570=1.0}, valueCount=1, labels={app=OB, ob_cluster_id=10, ob_cluster_name=darenOB, server_ips=133.197.204.3})]}, matched=true, message=null, level=2) 2023-11-27 13:39:45.107 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063585, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:45.107 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063585, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:45.107 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063585, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:45.121 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063585, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:45.122 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063585, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:45.250 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:39:45.287 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:39:45.368 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:39:45.759 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:39:46.165 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:39:46.166 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:39:46.356 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:39:49.166 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:39:49.168 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:39:50.103 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063590, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:50.103 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063590, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:50.103 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063590, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:50.104 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063590, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:50.106 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063590, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:51.559 INFO 10 --- [pool-ocp-schedules-5,,8ef7de44e45d] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-11-27 13:39:51.583 INFO 10 --- [pool-ocp-schedules-5,,8ef7de44e45d] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-11-27 13:39:51.698 INFO 10 --- [pool-ocp-schedules-9,,9a416eee1395] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":5121,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":1707,"seriesIdKeySize":87133,"secondWriteQueue":0,"minuteWriteCacheSize":32707} 2023-11-27 13:39:51.698 INFO 10 --- [pool-ocp-schedules-9,,9a416eee1395] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_zone_current_timestamp=38, ob_server_resource_memory_assigned_bytes=21, tcp_retrans=30, ob_unit_num=21, go_memstats_mspan_sys_bytes=17, go_memstats_stack_sys_bytes=17, exporter_export_metrics_bytes_total=28, node_filesystem_size_bytes=87, http_server_requests_seconds_count=40, ob_compaction_frozen_time=18, ob_system_event_total_waits=34, monagent_input_sql_duration_seconds_sum=60, observer_fd_count=20, ob_plan_cache_hit_total=28, processor_process_metrics_total=22, node_filesystem_readonly=86, ob_server_resource_disk_in_use_bytes=19, go_memstats_frees_total=17, sql_audit_input_collect_data_jumped_total=12, io_await=60, host_available=13, go_memstats_mallocs_total=17, mysql_output_write_sql_failed_count=23, ob_unit_config_max_memory_bytes=45, io_util=60, ob_system_event_detail_total_timeouts=782, ob_tenant_disk_v4_data_size=18, ob_latch_sleep_total=326, ob_tenant_unit_min_memory=41, ob_memstore_total_bytes=28, process_resident_memory_bytes=17, ob_cache_size_bytes=47, ob_tenant_unit_max_memory=41, ob_tenant_assigned_cpu_total=25, input_collect_metrics_total=29, ob_tenant_server_available_disk_free=28, ob_tenant_server_avg_cpu_used_percentage=23, node_disk_io_time_weighted_seconds_total=39, node_disk_write_time_seconds_total=39, cpu_count=30, output_write_metrics_total=15, ob_cluster_sync=175, process_open_fds=17, odp_transaction_total=22, ob_system_event_detail_time_waited_us=782, process_max_fds=17, ob_server_resource_cpu_assigned_percent=20, ocp_alarm_detect_duration_ms_count=82, go_memstats_heap_inuse_bytes=17, ob_tenant_server_max_cpu_used_percentage=23, proxy_memory_used_bytes=30, go_memstats_mcache_sys_bytes=17, ob_tenant_disk_v4_log_size=18, node_memory_memfree_bytes=29, process_exists=38, go_gc_duration_seconds=22, ob_server_resource_disk_bytes=21, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=27, node_memory_cached_bytes=29, ob_table_num=24, ob_disk_free_bytes=21, node_filesystem_avail_bytes=87, ob_backup_storage_capacity_bytes=349, ob_active_session_num=27, ob_unit_config_max_iops=45, ob_plan_cache_memory_bytes=28, node_network_transmit_errs_total=65, go_memstats_last_gc_time_seconds=17, go_gc_duration_seconds_count=17, monitor_exporter_avaliable=78, ob_tenant_server_max_memory_used_percentage=23, node_disk_read_bytes_total=39, mysql_output_write_sql_milliseconds_summary_count=25, ob_compaction_is_suspended=18, ob_tenant_unit_log_disk_size=22, go_memstats_lookups_total=17, mysql_output_write_sql_count=25, log_tailer_count=17, host_agent_process_status=21, ob_server_resource_cpu=21, go_memstats_alloc_bytes=17, ob_server_num=58, node_cpu_seconds_total_native=47, ob_tenant_unit_count=39, ob_session_all_num=25, ob_tenant_resource_min_memory=25, ob_memstore_freeze_times=28, ob_system_event_time_waited=34, ob_tenant_resource_min_cpu=25, ob_cluster_status_check=49, monagent_plugin_execute_seconds_total=41, node_net_bandwidth_bps=45, monagent_oceanbase_db_pool_idle=7, go_memstats_heap_idle_bytes=17, go_memstats_gc_sys_bytes=17, ob_server_resource_memory_bytes=21, ob_tenant_sys_partition_required_size=12, ob_latch_get_total=326, go_info=18, ob_tenant_assigned_mem_assigned=25, log_tailer_processed_file_count=24, node_load15=29, ob_tenant_server_required_size=35, ob_unit_config_min_iops=45, ob_server_partition_replica_limit=7, ob_tenant_unit_max_cpu=41, ob_latch_wait_seconds_total=326, node_memory_buffers_bytes=29, go_memstats_heap_released_bytes=17, node_network_receive_bytes_total=65, ob_memstore_freeze_trigger_bytes=28, ob_waitevent_wait_seconds_total=28, ob_server_stopped_duration_seconds=20, exposer_request_latencies_count=8, ob_role_is_rootserver=20, go_memstats_next_gc_bytes=17, mysql_output_context_timeout_discard_metrics_count=17, ob_session_active_num=25, node_filesystem_files_free=109, ob_memtable_max_snapshot_duration_seconds=27, node_disk_read_time_seconds_total=39, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=22, process_virtual_memory_bytes=17, mysql_output_write_sql_retry_count=23, ob_tenant_assigned_cpu_assigned=25, ob_tenant_unit_min_cpu=41, ob_unit_config_max_cpu=45, ob_unit_config_max_session_num=18, http_request_milliseconds_summary_count=79, monagent_oceanbase_audit_timeinfo_cache_size=7, net_connectivity=10, monagent_input_sql_duration_seconds_bucket=70, ob_latch_immediate_get_total=326, node_cpu_seconds_total=37, http_request_milliseconds_summary=82, ob_latch_immediate_miss_total=326, ob_index_error_num=16, ob_tenant_server_available_disk_total=28, ob_latch_spin_get_total=326, ob_tenant_memory_hold_bytes=18, go_memstats_buck_hash_sys_bytes=17, monagent_pipeline_report_metrics_total=18, mysql_output_write_sql_milliseconds_summary_sum=25, ob_tenant_disk_total_size=25, exposer_transferred_bytes_total=8, odp_sql_request_total=67, mysql_output_write_task_size=24, ob_tenant_context_memory_count=339, node_disk_reads_completed_total=39, exporter_export_metrics_total=28, ob_sysstat=129, ob_partition_num=13, node_ntp_offset_seconds=29, ob_latch_miss_total=326, go_memstats_heap_sys_bytes=17, ocp_meta_db_health=4, ob_disk_total_bytes=21, ob_zone_stat=38, go_memstats_heap_objects=17, go_memstats_mcache_inuse_bytes=17, go_memstats_other_sys_bytes=17, go_threads=17, http_request_milliseconds_summary_sum=79, ob_memstore_active_bytes=28, http_server_requests_seconds_sum=40, go_memstats_alloc_bytes_total=17, ob_tenant_server_partition_replica_count=35, ob_tenant_assigned_mem_total=25, node_memory_memtotal_bytes=29, node_network_transmit_drop_total=65, ob_system_event_detail_total_waits=782, odp_used_connections=28, ob_tenant_task_max_duration_seconds=25, backup_storage_capacity_used_percent=14, exposer_request_latencies=11, ob_waitevent_wait_total=28, odp_sql_cost_total=66, ob_tenant500_memory_used_bytes=20, node_network_transmit_bytes_total=65, ob_compaction_is_error=18, node_network_receive_drop_total=65, ob_server_partition_replica_count=7, monagent_pipeline_execute_total=27, node_filesystem_files=109, ob_tenant_server_avg_memory_used_percentage=23, process_start_time_seconds=17, go_memstats_heap_alloc_bytes=17, ob_tenant_resource_max_cpu=25, go_gc_duration_seconds_sum=17, go_memstats_mspan_inuse_bytes=17, mysql_output_write_sql_milliseconds_summary=28, ob_server_resource_memory_assigned_percent=20, monagent_pipeline_execute_seconds_total=27, go_memstats_sys_bytes=17, node_network_receive_errs_total=65, ob_all_session_num=27, observer_ulimit_max_fd_count=20, ob_plan_cache_access_total=28, ob_tenant_server_row_count=35, ob_tenant_memory_used_bytes=18, monagent_oceanbase_plan_cache_size=7, ob_tenant_server_data_size=35, go_memstats_gc_cpu_fraction=17, monagent_oceanbase_audit_cache_size=7, ob_tenant_context_memory_used_bytes=339, node_disk_writes_completed_total=39, node_load1=29, process_cpu_seconds_total=17, ob_tenant_resource_max_memory=25, go_memstats_stack_inuse_bytes=17, ob_tenant_user_partition_required_size=12, node_load5=29, monagent_input_sql_duration_seconds_count=60, node_disk_written_bytes_total=39, monagent_oceanbase_ash_cache_size=7, ob_tenant_context_memory_hold_bytes=339, go_goroutines=17, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=21, exposer_request_latencies_sum=8, ob_unit_config_min_cpu=45, process_virtual_memory_max_bytes=17, monagent_oceanbase_plan_timeinfo_cache_size=7, monagent_plugin_execute_total=41, ob_tenant500_memory_hold_bytes=20, exposer_scrapes_total=8, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=22, ob_compaction_start_time=18, oceanbase_connectivity=21} 2023-11-27 13:39:51.803 INFO 10 --- [pool-overview-container-4,,88537b888a33] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:39:51.922 INFO 10 --- [pool-scanner--12319-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=88 2023-11-27 13:39:52.169 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:39:52.173 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:39:52.237 INFO 10 --- [pool-ocp-schedules-19,,0cfcb738cf82] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-11-27 13:39:52.239 INFO 10 --- [pool-ocp-schedules-1,,78764e2e30f6] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-11-27 13:39:52.240 INFO 10 --- [pool-ocp-schedules-19,,0cfcb738cf82] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-11-27 13:39:52.241 INFO 10 --- [pool-ocp-schedules-1,,78764e2e30f6] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-11-27 13:39:52.247 INFO 10 --- [pool-ocp-schedules-5,,3b35caba83fe] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-11-27 13:39:52.249 INFO 10 --- [pool-ocp-schedules-5,,3b35caba83fe] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-11-27 13:39:52.319 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:52.321 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:52.321 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:52.322 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:52.322 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:52.322 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:52.322 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:52.323 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:52.323 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:52.323 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:52.324 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, MAX(CASE `name` WHEN 'region' THEN `info` ELSE '' END ) `region`, MAX(CASE `name` WHEN 'idc' THEN `info` ELSE '' END ) `idc`, MAX(CASE `name` WHEN 'status' THEN `info` ELSE '' END ) `status`, MAX(CASE `name` WHEN 'merge_status' THEN `info` ELSE '' END ) `merge_status`, MAX(CASE `name` WHEN 'broadcast_version' THEN `value` ELSE 0 END ) `broadcast_version`, MAX(CASE `name` WHEN 'all_merged_version' THEN `value` ELSE 0 END ) `all_merged_version`, MAX(CASE `name` WHEN 'last_merged_version' THEN `value` ELSE 0 END ) `last_merged_version`, MAX(CASE `name` WHEN 'merge_start_time' THEN `value` ELSE 0 END ) `merge_start_time`, MAX(CASE `name` WHEN 'last_merged_time' THEN `value` ELSE 0 END ) `last_merged_time`, MAX(CASE `name` WHEN 'is_merge_timeout' THEN `value` ELSE 0 END ) `merge_timeout` FROM oceanbase.__all_zone WHERE `zone` <> '' GROUP BY `zone` 2023-11-27 13:39:52.327 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:52.327 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:52.334 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-11-27 13:39:52.339 INFO 10 --- [ForkJoinPool-38-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:52.340 INFO 10 --- [ForkJoinPool-38-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:52.340 INFO 10 --- [ForkJoinPool-38-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:52.341 INFO 10 --- [ForkJoinPool-38-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:52.345 INFO 10 --- [ForkJoinPool-38-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:52.345 INFO 10 --- [ForkJoinPool-38-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:52.346 INFO 10 --- [ForkJoinPool-38-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:52.348 INFO 10 --- [ForkJoinPool-38-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-11-27 13:39:52.350 INFO 10 --- [ForkJoinPool-38-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:52.351 INFO 10 --- [ForkJoinPool-38-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:52.351 INFO 10 --- [ForkJoinPool-38-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:52.352 INFO 10 --- [ForkJoinPool-38-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:52.355 INFO 10 --- [ForkJoinPool-38-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:52.355 INFO 10 --- [ForkJoinPool-38-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:52.357 INFO 10 --- [ForkJoinPool-38-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:52.359 INFO 10 --- [ForkJoinPool-38-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=3.1.4 2023-11-27 13:39:52.468 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:39:52.469 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:52.469 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:52.469 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:52.469 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:52.471 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:52.471 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:52.471 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, MAX(CASE `name` WHEN 'region' THEN `info` ELSE '' END ) `region`, MAX(CASE `name` WHEN 'idc' THEN `info` ELSE '' END ) `idc`, MAX(CASE `name` WHEN 'status' THEN `info` ELSE '' END ) `status`, MAX(CASE `name` WHEN 'merge_status' THEN `info` ELSE '' END ) `merge_status`, MAX(CASE `name` WHEN 'broadcast_version' THEN `value` ELSE 0 END ) `broadcast_version`, MAX(CASE `name` WHEN 'all_merged_version' THEN `value` ELSE 0 END ) `all_merged_version`, MAX(CASE `name` WHEN 'last_merged_version' THEN `value` ELSE 0 END ) `last_merged_version`, MAX(CASE `name` WHEN 'merge_start_time' THEN `value` ELSE 0 END ) `merge_start_time`, MAX(CASE `name` WHEN 'last_merged_time' THEN `value` ELSE 0 END ) `last_merged_time`, MAX(CASE `name` WHEN 'is_merge_timeout' THEN `value` ELSE 0 END ) `merge_timeout` FROM oceanbase.__all_zone WHERE `zone` <> '' GROUP BY `zone` 2023-11-27 13:39:52.472 WARN 10 --- [ForkJoinPool-37-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-11-27 13:39:52.473 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:52.473 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:52.473 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, stop_time, start_service_time, block_migrate_in_time, id FROM oceanbase.__all_server ORDER BY zone ASC, id ASC 2023-11-27 13:39:52.475 INFO 10 --- [ForkJoinPool-37-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.206.97:2882, status=ACTIVE, startServiceTime=1701063018577428, active=true 2023-11-27 13:39:52.475 INFO 10 --- [ForkJoinPool-37-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=133.197.206.97:2882, stopTime=0, stopped=false 2023-11-27 13:39:52.475 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=IDLE, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=IDLE, startTime=null, stopTime=null} 2023-11-27 13:39:52.506 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=IDLE, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:39:52.506 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=133.197.206.97, port=2882, sqlPort=2881, version=3.1.4_10000092022071511-b4bfa011ceaef428782dcb65ae89190c40b78c2f(Jul 15 2022 11:45:14), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-27T13:30:19+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.206.97, port=2882, sqlPort=2881, version=3.1.4_10000092022071511-b4bfa011ceaef428782dcb65ae89190c40b78c2f(Jul 15 2022 11:45:14), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-27T05:30:18.577428Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:39:52.523 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=133.197.206.97, port=2882, sqlPort=2881, version=3.1.4_10000092022071511-b4bfa011ceaef428782dcb65ae89190c40b78c2f(Jul 15 2022 11:45:14), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-27T05:30:18.577428Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:39:52.535 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:39:52.539 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:52.539 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:52.539 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:52.540 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:52.541 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:52.542 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:52.546 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:52.575 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:39:52.576 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:52.576 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:52.576 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:52.576 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:52.577 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:52.578 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:52.578 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:52.585 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:52.585 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:52.587 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT cluster_id, cluster_name, created, cluster_role, cluster_status, current_scn, standby_became_primary_scn, primary_cluster_id, (time_to_usec(now(6)) - current_scn)/1000000 as delay_time, protection_mode, redo_transport_options, protection_level FROM oceanbase.v$ob_cluster 2023-11-27 13:39:52.594 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:52.594 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:52.594 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT switchover_status, switchover_info FROM oceanbase.v$ob_cluster 2023-11-27 13:39:52.604 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=obcluster, obClusterId=1 2023-11-27 13:39:52.604 INFO 10 --- [ForkJoinPool-37-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"133.197.206.97:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1701063583198000}, createTime=2022-11-29T14:52:53+08:00, creator=admin, obVersion=3.1.4, tenantCount=3) 2023-11-27 13:39:52.841 INFO 10 --- [http-nio-0.0.0.0-8080-exec-6,0ae1f89c39ef45b2,098f87a4e59e] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.10, traceId=0ae1f89c39ef45b2, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:39:52.864 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:39:52.860767+08:00, lastProcessTime=2023-11-27T13:39:42.821243+08:00 2023-11-27 13:39:52.911 INFO 10 --- [http-nio-0.0.0.0-8080-exec-6,0ae1f89c39ef45b2,098f87a4e59e] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.10, traceId=0ae1f89c39ef45b2, duration=71 ms] 2023-11-27 13:39:53.066 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:39:52.860767+08:00 2023-11-27 13:39:53.181 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:39:53.183 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:53.183 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:53.183 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:53.184 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:53.190 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:53.190 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:53.191 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-11-27 13:39:53.194 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:53.194 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:53.195 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-11-27 13:39:53.197 INFO 10 --- [ForkJoinPool-37-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.204.3:2882, status=INACTIVE, startServiceTime=0, active=false 2023-11-27 13:39:53.197 INFO 10 --- [ForkJoinPool-37-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.204.10:2882, status=ACTIVE, startServiceTime=1695730092226557, active=true 2023-11-27 13:39:53.197 INFO 10 --- [ForkJoinPool-37-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=133.197.204.10:2882, stopTime=0, stopped=false 2023-11-27 13:39:53.198 INFO 10 --- [ForkJoinPool-37-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.204.11:2882, status=ACTIVE, startServiceTime=1695730305487530, active=true 2023-11-27 13:39:53.198 INFO 10 --- [ForkJoinPool-37-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=133.197.204.11:2882, stopTime=0, stopped=false 2023-11-27 13:39:53.198 INFO 10 --- [ForkJoinPool-37-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.204.12:2882, status=ACTIVE, startServiceTime=1695730736949667, active=true 2023-11-27 13:39:53.198 INFO 10 --- [ForkJoinPool-37-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=133.197.204.12:2882, stopTime=0, stopped=false 2023-11-27 13:39:53.198 INFO 10 --- [ForkJoinPool-37-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.204.13:2882, status=ACTIVE, startServiceTime=1695731005656813, active=true 2023-11-27 13:39:53.198 INFO 10 --- [ForkJoinPool-37-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=133.197.204.13:2882, stopTime=0, stopped=false 2023-11-27 13:39:53.198 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=14, name=zone6, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone6, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-27 13:39:53.267 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=14, name=zone6, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:39:53.267 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=14, ip=133.197.204.13, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=7, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T20:23:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.204.13, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=7, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:23:25.656813Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:39:53.296 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=14, ip=133.197.204.13, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=7, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:23:25.656813Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:39:53.296 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=13, name=zone5, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone5, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-27 13:39:53.324 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=13, name=zone5, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:39:53.325 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=13, ip=133.197.204.12, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=6, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T20:18:57+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.204.12, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=6, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:18:56.949667Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:39:53.357 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=13, ip=133.197.204.12, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=6, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:18:56.949667Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:39:53.358 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=12, name=zone4, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone4, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-27 13:39:53.382 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=12, name=zone4, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:39:53.382 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=12, ip=133.197.204.11, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T20:11:45+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.204.11, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:11:45.487530Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:39:53.409 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=12, ip=133.197.204.11, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:11:45.487530Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:39:53.409 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=11, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-27 13:39:53.426 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=11, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:39:53.426 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=11, ip=133.197.204.10, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=true, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T20:08:12+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.204.10, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=true, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:08:12.226557Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:39:53.446 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=11, ip=133.197.204.10, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=true, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:08:12.226557Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:39:53.446 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-27 13:39:53.477 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:39:53.477 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=133.197.204.3, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=2, status=UNAVAILABLE, innerStatus=INACTIVE, startTime=1970-01-01T08:00+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.204.3, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=2, status=null, innerStatus=INACTIVE, startTime=1970-01-01T00:00Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:39:53.516 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=133.197.204.3, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=2, status=UNAVAILABLE, innerStatus=INACTIVE, startTime=1970-01-01T00:00Z, stopTime=1970-01-01T00:00Z}, oldStatus=UNAVAILABLE, newStatus=UNAVAILABLE 2023-11-27 13:39:53.537 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:39:53.538 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:53.538 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:53.538 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:53.539 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:53.540 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.120.0.1:8080 2023-11-27 13:39:53.544 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:53.544 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:53.545 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:53.576 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=darenOB, obClusterId=10 2023-11-27 13:39:53.577 INFO 10 --- [ForkJoinPool-37-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=darenOB, obClusterId=10, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"darenOB","ObClusterId":10,"ObRegion":"darenOB","ObRegionId":10,"ReadonlyRsList":[],"RsList":[{"address":"133.197.204.10:2882","role":"LEADER","sql_port":2881},{"address":"133.197.204.11:2882","role":"FOLLOWER","sql_port":2881},{"address":"133.197.204.12:2882","role":"FOLLOWER","sql_port":2881},{"address":"133.197.204.13:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1701063480849000}, createTime=2022-11-29T15:05:37+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-11-27 13:39:53.681 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.120.0.1:8080 2023-11-27 13:39:53.931 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.120.0.1:8080 2023-11-27 13:39:54.236 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:39:54.263 INFO 10 --- [http-nio-0.0.0.0-8080-exec-1,44378bc2ed844e75,2373d16fee77] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.12, traceId=44378bc2ed844e75, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:39:54.300 INFO 10 --- [http-nio-0.0.0.0-8080-exec-1,44378bc2ed844e75,2373d16fee77] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.12, traceId=44378bc2ed844e75, duration=38 ms] 2023-11-27 13:39:54.522 INFO 10 --- [alarm-detect-15,,d3ba129eaaaf] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_cluster_exists_inactive_server detect time range 1701063574 ~ 1701063584 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={inactive_server_count > 0=[ExpressionEvaluateResult(matchValue=0.0, metric=inactive_server_count, match=true, value=1.0, values={1701063580=1.0}, valueCount=1, labels={app=OB, ob_cluster_id=10, ob_cluster_name=darenOB, server_ips=133.197.204.3})]}, matched=true, message=null, level=2) 2023-11-27 13:39:54.681 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.120.0.1:8080 2023-11-27 13:39:54.764 INFO 10 --- [pool-ocp-schedules-18,,24b0afc992f2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:39:54.940 INFO 10 --- [http-nio-0.0.0.0-8080-exec-3,34c9f581d22d48fd,daf88d32c019] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.12, traceId=34c9f581d22d48fd, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:39:54.960 INFO 10 --- [http-nio-0.0.0.0-8080-exec-3,34c9f581d22d48fd,daf88d32c019] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.12, traceId=34c9f581d22d48fd, duration=20 ms] 2023-11-27 13:39:55.011 INFO 10 --- [http-nio-0.0.0.0-8080-exec-5,ee7dd567ccc241b6,e664032a809b] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.10, traceId=ee7dd567ccc241b6, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:39:55.022 INFO 10 --- [http-nio-0.0.0.0-8080-exec-5,ee7dd567ccc241b6,e664032a809b] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.10, traceId=ee7dd567ccc241b6, duration=11 ms] 2023-11-27 13:39:55.103 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063595, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:55.103 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063595, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:55.103 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063595, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:55.104 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063595, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:55.123 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063595, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:39:55.173 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:39:55.174 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:39:55.288 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:39:55.290 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:39:55.396 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:39:55.766 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:39:55.796 INFO 10 --- [http-nio-0.0.0.0-8080-exec-7,827fb5a8aef74aa5,1aa6297d45cf] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.11, traceId=827fb5a8aef74aa5, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:39:55.802 INFO 10 --- [http-nio-0.0.0.0-8080-exec-7,827fb5a8aef74aa5,1aa6297d45cf] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.11, traceId=827fb5a8aef74aa5, duration=6 ms] 2023-11-27 13:39:56.015 INFO 10 --- [pool-ocp-schedules-3,,346b45d640e9] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:39:56.015 INFO 10 --- [pool-ocp-schedules-3,,346b45d640e9] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:39:56.104 INFO 10 --- [pool-scanner--12412-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=88 2023-11-27 13:39:56.162 INFO 10 --- [pool-scanner--12413-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=88 2023-11-27 13:39:56.262 INFO 10 --- [pool-scanner--12414-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=88 2023-11-27 13:39:56.350 INFO 10 --- [pool-ocp-schedules-4,,ef4b3fc2eeda] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:39:56.362 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:39:56.389 INFO 10 --- [pool-ocp-schedules-4,,ef4b3fc2eeda] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:39:56.530 INFO 10 --- [pool-ocp-schedules-4,,ef4b3fc2eeda] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:39:56.559 INFO 10 --- [pool-ocp-schedules-4,,ef4b3fc2eeda] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:39:56.600 INFO 10 --- [pool-ocp-schedules-4,,ef4b3fc2eeda] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:39:56.629 INFO 10 --- [pool-ocp-schedules-4,,ef4b3fc2eeda] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:39:58.178 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:39:58.181 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:39:58.907 INFO 10 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2023-11-27 13:39:58.996 INFO 10 --- [sql-rollup-daemon,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:39:58.997 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:58.997 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:58.997 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:58.998 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:58.999 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:39:58.999 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:58.999 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, stop_time, start_service_time, block_migrate_in_time, id FROM oceanbase.__all_server ORDER BY zone ASC, id ASC 2023-11-27 13:39:59.027 INFO 10 --- [pool-prometheus-exporter-check2,,49b16e74c908] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/stat, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:39:59.027 WARN 10 --- [pool-prometheus-exporter-check2,,49b16e74c908] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/stat is inactive 2023-11-27 13:39:59.028 INFO 10 --- [pool-prometheus-exporter-check2,,49b16e74c908] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62888/metrics/stat, message=java.net.ConnectException: Connection refused: /133.197.204.3:62888 2023-11-27 13:39:59.028 WARN 10 --- [pool-prometheus-exporter-check2,,49b16e74c908] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62888/metrics/stat is inactive 2023-11-27 13:39:59.032 INFO 10 --- [pool-prometheus-exporter-check2,,49b16e74c908] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/node/host, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:39:59.032 WARN 10 --- [pool-prometheus-exporter-check2,,49b16e74c908] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/node/host is inactive 2023-11-27 13:39:59.033 INFO 10 --- [pool-prometheus-exporter-check2,,49b16e74c908] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/obproxy, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:39:59.033 WARN 10 --- [pool-prometheus-exporter-check2,,49b16e74c908] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/obproxy is inactive 2023-11-27 13:39:59.035 INFO 10 --- [pool-prometheus-exporter-check2,,49b16e74c908] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/node/obproxy, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:39:59.035 WARN 10 --- [pool-prometheus-exporter-check2,,49b16e74c908] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/node/obproxy is inactive 2023-11-27 13:39:59.040 INFO 10 --- [pool-prometheus-exporter-check2,,49b16e74c908] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/ob/extra, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:39:59.041 WARN 10 --- [pool-prometheus-exporter-check2,,49b16e74c908] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/ob/extra is inactive 2023-11-27 13:39:59.042 INFO 10 --- [pool-prometheus-exporter-check2,,49b16e74c908] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/node/ob, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:39:59.042 WARN 10 --- [pool-prometheus-exporter-check2,,49b16e74c908] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/node/ob is inactive 2023-11-27 13:39:59.048 INFO 10 --- [pool-prometheus-exporter-check2,,49b16e74c908] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/ob/basic, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:39:59.048 WARN 10 --- [pool-prometheus-exporter-check2,,49b16e74c908] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/ob/basic is inactive 2023-11-27 13:39:59.070 INFO 10 --- [sql-rollup-daemon,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:39:59.076 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:59.081 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:59.081 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:59.084 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:39:59.088 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:39:59.088 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:59.089 INFO 10 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-11-27 13:39:59.094 INFO 10 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (darenOB:10-3) 2023-11-27 13:39:59.094 INFO 10 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-1) 2023-11-27 13:39:59.094 INFO 10 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (darenOB:10-4) 2023-11-27 13:39:59.094 INFO 10 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (darenOB:10-1) 2023-11-27 13:39:59.094 INFO 10 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (darenOB:10-5) 2023-11-27 13:39:59.094 INFO 10 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (darenOB:10-6) 2023-11-27 13:39:59.106 INFO 10 --- [sql-rollup-worker-4,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:39:59.106 INFO 10 --- [sql-rollup-worker-4,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:59.108 INFO 10 --- [sql-rollup-worker-5,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:39:59.108 INFO 10 --- [sql-rollup-worker-5,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:59.109 INFO 10 --- [sql-rollup-worker-4,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:39:59.110 INFO 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=1, level=0->1, c=0) - not enough cached interval (source=2023-04-25T14:52:31, target=2023-04-25T14:50:31, granule=120000ms, delay=120000ms) 2023-11-27 13:39:59.110 INFO 10 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=1, level=1->2, c=0) - not enough cached interval (source=2023-04-25T14:50:31, target=2023-04-25T14:46:01, granule=600000ms, delay=270000ms) 2023-11-27 13:39:59.111 INFO 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=darenOB:10, server=1, level=0->1, c=0) - not enough cached interval (source=2023-11-24T10:14:01, target=2023-11-24T10:11:53, granule=120000ms, delay=128000ms) 2023-11-27 13:39:59.111 INFO 10 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=darenOB:10, server=1, level=1->2, c=0) - not enough cached interval (source=2023-11-24T10:11:53, target=2023-11-24T10:05:53, granule=600000ms, delay=360000ms) 2023-11-27 13:39:59.111 INFO 10 --- [sql-rollup-worker-5,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:39:59.113 INFO 10 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:39:59.113 INFO 10 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:59.114 INFO 10 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:39:59.114 INFO 10 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:39:59.114 INFO 10 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:39:59.117 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=0) - no updates (start=2023-11-26T17:46:00, end=2023-11-26T17:48:30) 2023-11-27 13:39:59.117 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=0) - no updates (start=2023-11-26T17:46:00, end=2023-11-26T17:48:30) 2023-11-27 13:39:59.119 INFO 10 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:39:59.138 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=0) - no updates (start=2023-11-26T17:46:00, end=2023-11-26T17:48:30) 2023-11-27 13:39:59.140 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=0) - no updates (start=2023-11-26T17:46:00, end=2023-11-26T17:48:30) 2023-11-27 13:39:59.146 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=1) - no updates (start=2023-11-26T17:48:30, end=2023-11-26T17:51:00) 2023-11-27 13:39:59.147 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=1) - no updates (start=2023-11-26T17:48:30, end=2023-11-26T17:51:00) 2023-11-27 13:39:59.153 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=1) - no updates (start=2023-11-26T17:48:30, end=2023-11-26T17:51:00) 2023-11-27 13:39:59.155 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=1) - no updates (start=2023-11-26T17:48:30, end=2023-11-26T17:51:00) 2023-11-27 13:39:59.159 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=2) - no updates (start=2023-11-26T17:51:00, end=2023-11-26T17:53:30) 2023-11-27 13:39:59.162 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=2) - no updates (start=2023-11-26T17:51:00, end=2023-11-26T17:53:30) 2023-11-27 13:39:59.162 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=2) - no updates (start=2023-11-26T17:51:00, end=2023-11-26T17:53:30) 2023-11-27 13:39:59.163 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=3) - no updates (start=2023-11-26T17:53:30, end=2023-11-26T17:56:00) 2023-11-27 13:39:59.166 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=2) - no updates (start=2023-11-26T17:51:00, end=2023-11-26T17:53:30) 2023-11-27 13:39:59.170 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=3) - no updates (start=2023-11-26T17:53:30, end=2023-11-26T17:56:00) 2023-11-27 13:39:59.171 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=3) - no updates (start=2023-11-26T17:53:30, end=2023-11-26T17:56:00) 2023-11-27 13:39:59.171 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=4) - no updates (start=2023-11-26T17:56:00, end=2023-11-26T17:58:30) 2023-11-27 13:39:59.181 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=3) - no updates (start=2023-11-26T17:53:30, end=2023-11-26T17:56:00) 2023-11-27 13:39:59.184 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=5) - no updates (start=2023-11-26T17:58:30, end=2023-11-26T18:01:00) 2023-11-27 13:39:59.198 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=4) - no updates (start=2023-11-26T17:56:00, end=2023-11-26T17:58:30) 2023-11-27 13:39:59.198 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=4) - no updates (start=2023-11-26T17:56:00, end=2023-11-26T17:58:30) 2023-11-27 13:39:59.198 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=4) - no updates (start=2023-11-26T17:56:00, end=2023-11-26T17:58:30) 2023-11-27 13:39:59.204 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=6) - no updates (start=2023-11-26T18:01:00, end=2023-11-26T18:03:30) 2023-11-27 13:39:59.209 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=5) - no updates (start=2023-11-26T17:58:30, end=2023-11-26T18:01:00) 2023-11-27 13:39:59.209 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=5) - no updates (start=2023-11-26T17:58:30, end=2023-11-26T18:01:00) 2023-11-27 13:39:59.209 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=7) - no updates (start=2023-11-26T18:03:30, end=2023-11-26T18:06:00) 2023-11-27 13:39:59.209 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=5) - no updates (start=2023-11-26T17:58:30, end=2023-11-26T18:01:00) 2023-11-27 13:39:59.213 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=6) - no updates (start=2023-11-26T18:01:00, end=2023-11-26T18:03:30) 2023-11-27 13:39:59.217 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=6) - no updates (start=2023-11-26T18:01:00, end=2023-11-26T18:03:30) 2023-11-27 13:39:59.217 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=6) - no updates (start=2023-11-26T18:01:00, end=2023-11-26T18:03:30) 2023-11-27 13:39:59.217 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=8) - no updates (start=2023-11-26T18:06:00, end=2023-11-26T18:08:30) 2023-11-27 13:39:59.221 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=7) - no updates (start=2023-11-26T18:03:30, end=2023-11-26T18:06:00) 2023-11-27 13:39:59.229 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=9) - no updates (start=2023-11-26T18:08:30, end=2023-11-26T18:11:00) 2023-11-27 13:39:59.235 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=7) - no updates (start=2023-11-26T18:03:30, end=2023-11-26T18:06:00) 2023-11-27 13:39:59.241 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=10) - no updates (start=2023-11-26T18:11:00, end=2023-11-26T18:13:30) 2023-11-27 13:39:59.243 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=7) - no updates (start=2023-11-26T18:03:30, end=2023-11-26T18:06:00) 2023-11-27 13:39:59.249 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=11) - no updates (start=2023-11-26T18:13:30, end=2023-11-26T18:16:00) 2023-11-27 13:39:59.256 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=8) - no updates (start=2023-11-26T18:06:00, end=2023-11-26T18:08:30) 2023-11-27 13:39:59.259 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=12) - no updates (start=2023-11-26T18:16:00, end=2023-11-26T18:18:30) 2023-11-27 13:39:59.261 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=9) - no updates (start=2023-11-26T18:08:30, end=2023-11-26T18:11:00) 2023-11-27 13:39:59.263 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=13) - no updates (start=2023-11-26T18:18:30, end=2023-11-26T18:21:00) 2023-11-27 13:39:59.265 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=8) - no updates (start=2023-11-26T18:06:00, end=2023-11-26T18:08:30) 2023-11-27 13:39:59.266 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=10) - no updates (start=2023-11-26T18:11:00, end=2023-11-26T18:13:30) 2023-11-27 13:39:59.269 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=14) - no updates (start=2023-11-26T18:21:00, end=2023-11-26T18:23:30) 2023-11-27 13:39:59.271 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=11) - no updates (start=2023-11-26T18:13:30, end=2023-11-26T18:16:00) 2023-11-27 13:39:59.280 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=15) - no updates (start=2023-11-26T18:23:30, end=2023-11-26T18:26:00) 2023-11-27 13:39:59.284 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=12) - no updates (start=2023-11-26T18:16:00, end=2023-11-26T18:18:30) 2023-11-27 13:39:59.289 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=9) - no updates (start=2023-11-26T18:08:30, end=2023-11-26T18:11:00) 2023-11-27 13:39:59.291 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=8) - no updates (start=2023-11-26T18:06:00, end=2023-11-26T18:08:30) 2023-11-27 13:39:59.300 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=16) - no updates (start=2023-11-26T18:26:00, end=2023-11-26T18:28:30) 2023-11-27 13:39:59.305 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=13) - no updates (start=2023-11-26T18:18:30, end=2023-11-26T18:21:00) 2023-11-27 13:39:59.306 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=10) - no updates (start=2023-11-26T18:11:00, end=2023-11-26T18:13:30) 2023-11-27 13:39:59.309 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=9) - no updates (start=2023-11-26T18:08:30, end=2023-11-26T18:11:00) 2023-11-27 13:39:59.311 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=17) - no updates (start=2023-11-26T18:28:30, end=2023-11-26T18:31:00) 2023-11-27 13:39:59.313 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=11) - no updates (start=2023-11-26T18:13:30, end=2023-11-26T18:16:00) 2023-11-27 13:39:59.320 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=10) - no updates (start=2023-11-26T18:11:00, end=2023-11-26T18:13:30) 2023-11-27 13:39:59.320 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=18) - no updates (start=2023-11-26T18:31:00, end=2023-11-26T18:33:30) 2023-11-27 13:39:59.322 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=12) - no updates (start=2023-11-26T18:16:00, end=2023-11-26T18:18:30) 2023-11-27 13:39:59.331 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=14) - no updates (start=2023-11-26T18:21:00, end=2023-11-26T18:23:30) 2023-11-27 13:39:59.335 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=11) - no updates (start=2023-11-26T18:13:30, end=2023-11-26T18:16:00) 2023-11-27 13:39:59.336 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=19) - no updates (start=2023-11-26T18:33:30, end=2023-11-26T18:36:00) 2023-11-27 13:39:59.351 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=13) - no updates (start=2023-11-26T18:18:30, end=2023-11-26T18:21:00) 2023-11-27 13:39:59.354 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=12) - no updates (start=2023-11-26T18:16:00, end=2023-11-26T18:18:30) 2023-11-27 13:39:59.364 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=14) - no updates (start=2023-11-26T18:21:00, end=2023-11-26T18:23:30) 2023-11-27 13:39:59.365 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=13) - no updates (start=2023-11-26T18:18:30, end=2023-11-26T18:21:00) 2023-11-27 13:39:59.366 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=15) - no updates (start=2023-11-26T18:23:30, end=2023-11-26T18:26:00) 2023-11-27 13:39:59.366 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=20) - no updates (start=2023-11-26T18:36:00, end=2023-11-26T18:38:30) 2023-11-27 13:39:59.387 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=14) - no updates (start=2023-11-26T18:21:00, end=2023-11-26T18:23:30) 2023-11-27 13:39:59.387 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=15) - no updates (start=2023-11-26T18:23:30, end=2023-11-26T18:26:00) 2023-11-27 13:39:59.390 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=16) - no updates (start=2023-11-26T18:26:00, end=2023-11-26T18:28:30) 2023-11-27 13:39:59.401 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=21) - no updates (start=2023-11-26T18:38:30, end=2023-11-26T18:41:00) 2023-11-27 13:39:59.407 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=15) - no updates (start=2023-11-26T18:23:30, end=2023-11-26T18:26:00) 2023-11-27 13:39:59.410 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=17) - no updates (start=2023-11-26T18:28:30, end=2023-11-26T18:31:00) 2023-11-27 13:39:59.411 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=16) - no updates (start=2023-11-26T18:26:00, end=2023-11-26T18:28:30) 2023-11-27 13:39:59.411 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=22) - no updates (start=2023-11-26T18:41:00, end=2023-11-26T18:43:30) 2023-11-27 13:39:59.415 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=17) - no updates (start=2023-11-26T18:28:30, end=2023-11-26T18:31:00) 2023-11-27 13:39:59.421 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=18) - no updates (start=2023-11-26T18:31:00, end=2023-11-26T18:33:30) 2023-11-27 13:39:59.422 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=18) - no updates (start=2023-11-26T18:31:00, end=2023-11-26T18:33:30) 2023-11-27 13:39:59.422 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=23) - no updates (start=2023-11-26T18:43:30, end=2023-11-26T18:46:00) 2023-11-27 13:39:59.427 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=16) - no updates (start=2023-11-26T18:26:00, end=2023-11-26T18:28:30) 2023-11-27 13:39:59.441 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=19) - no updates (start=2023-11-26T18:33:30, end=2023-11-26T18:36:00) 2023-11-27 13:39:59.448 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=19) - no updates (start=2023-11-26T18:33:30, end=2023-11-26T18:36:00) 2023-11-27 13:39:59.448 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=24) - no updates (start=2023-11-26T18:46:00, end=2023-11-26T18:48:30) 2023-11-27 13:39:59.453 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=17) - no updates (start=2023-11-26T18:28:30, end=2023-11-26T18:31:00) 2023-11-27 13:39:59.458 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=20) - no updates (start=2023-11-26T18:36:00, end=2023-11-26T18:38:30) 2023-11-27 13:39:59.462 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=20) - no updates (start=2023-11-26T18:36:00, end=2023-11-26T18:38:30) 2023-11-27 13:39:59.462 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=18) - no updates (start=2023-11-26T18:31:00, end=2023-11-26T18:33:30) 2023-11-27 13:39:59.464 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=21) - no updates (start=2023-11-26T18:38:30, end=2023-11-26T18:41:00) 2023-11-27 13:39:59.467 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=21) - no updates (start=2023-11-26T18:38:30, end=2023-11-26T18:41:00) 2023-11-27 13:39:59.469 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=19) - no updates (start=2023-11-26T18:33:30, end=2023-11-26T18:36:00) 2023-11-27 13:39:59.471 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=22) - no updates (start=2023-11-26T18:41:00, end=2023-11-26T18:43:30) 2023-11-27 13:39:59.471 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=22) - no updates (start=2023-11-26T18:41:00, end=2023-11-26T18:43:30) 2023-11-27 13:39:59.481 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=25) - no updates (start=2023-11-26T18:48:30, end=2023-11-26T18:51:00) 2023-11-27 13:39:59.483 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=20) - no updates (start=2023-11-26T18:36:00, end=2023-11-26T18:38:30) 2023-11-27 13:39:59.487 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=23) - no updates (start=2023-11-26T18:43:30, end=2023-11-26T18:46:00) 2023-11-27 13:39:59.490 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=26) - no updates (start=2023-11-26T18:51:00, end=2023-11-26T18:53:30) 2023-11-27 13:39:59.493 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=24) - no updates (start=2023-11-26T18:46:00, end=2023-11-26T18:48:30) 2023-11-27 13:39:59.493 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=23) - no updates (start=2023-11-26T18:43:30, end=2023-11-26T18:46:00) 2023-11-27 13:39:59.495 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=27) - no updates (start=2023-11-26T18:53:30, end=2023-11-26T18:56:00) 2023-11-27 13:39:59.503 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=21) - no updates (start=2023-11-26T18:38:30, end=2023-11-26T18:41:00) 2023-11-27 13:39:59.507 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=28) - no updates (start=2023-11-26T18:56:00, end=2023-11-26T18:58:30) 2023-11-27 13:39:59.508 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=24) - no updates (start=2023-11-26T18:46:00, end=2023-11-26T18:48:30) 2023-11-27 13:39:59.508 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=25) - no updates (start=2023-11-26T18:48:30, end=2023-11-26T18:51:00) 2023-11-27 13:39:59.512 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=22) - no updates (start=2023-11-26T18:41:00, end=2023-11-26T18:43:30) 2023-11-27 13:39:59.513 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=29) - no updates (start=2023-11-26T18:58:30, end=2023-11-26T19:01:00) 2023-11-27 13:39:59.513 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=25) - no updates (start=2023-11-26T18:48:30, end=2023-11-26T18:51:00) 2023-11-27 13:39:59.513 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=26) - no updates (start=2023-11-26T18:51:00, end=2023-11-26T18:53:30) 2023-11-27 13:39:59.517 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=26) - no updates (start=2023-11-26T18:51:00, end=2023-11-26T18:53:30) 2023-11-27 13:39:59.517 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=23) - no updates (start=2023-11-26T18:43:30, end=2023-11-26T18:46:00) 2023-11-27 13:39:59.518 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=30) - no updates (start=2023-11-26T19:01:00, end=2023-11-26T19:03:30) 2023-11-27 13:39:59.518 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=27) - no updates (start=2023-11-26T18:53:30, end=2023-11-26T18:56:00) 2023-11-27 13:39:59.520 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=24) - no updates (start=2023-11-26T18:46:00, end=2023-11-26T18:48:30) 2023-11-27 13:39:59.522 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=28) - no updates (start=2023-11-26T18:56:00, end=2023-11-26T18:58:30) 2023-11-27 13:39:59.522 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=27) - no updates (start=2023-11-26T18:53:30, end=2023-11-26T18:56:00) 2023-11-27 13:39:59.524 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=31) - no updates (start=2023-11-26T19:03:30, end=2023-11-26T19:06:00) 2023-11-27 13:39:59.534 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=28) - no updates (start=2023-11-26T18:56:00, end=2023-11-26T18:58:30) 2023-11-27 13:39:59.536 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=25) - no updates (start=2023-11-26T18:48:30, end=2023-11-26T18:51:00) 2023-11-27 13:39:59.537 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=29) - no updates (start=2023-11-26T18:58:30, end=2023-11-26T19:01:00) 2023-11-27 13:39:59.538 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=32) - no updates (start=2023-11-26T19:06:00, end=2023-11-26T19:08:30) 2023-11-27 13:39:59.541 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=26) - no updates (start=2023-11-26T18:51:00, end=2023-11-26T18:53:30) 2023-11-27 13:39:59.545 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=30) - no updates (start=2023-11-26T19:01:00, end=2023-11-26T19:03:30) 2023-11-27 13:39:59.546 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=27) - no updates (start=2023-11-26T18:53:30, end=2023-11-26T18:56:00) 2023-11-27 13:39:59.546 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=33) - no updates (start=2023-11-26T19:08:30, end=2023-11-26T19:11:00) 2023-11-27 13:39:59.547 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=29) - no updates (start=2023-11-26T18:58:30, end=2023-11-26T19:01:00) 2023-11-27 13:39:59.550 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=34) - no updates (start=2023-11-26T19:11:00, end=2023-11-26T19:13:30) 2023-11-27 13:39:59.552 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=28) - no updates (start=2023-11-26T18:56:00, end=2023-11-26T18:58:30) 2023-11-27 13:39:59.554 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=31) - no updates (start=2023-11-26T19:03:30, end=2023-11-26T19:06:00) 2023-11-27 13:39:59.554 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=35) - no updates (start=2023-11-26T19:13:30, end=2023-11-26T19:16:00) 2023-11-27 13:39:59.556 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=30) - no updates (start=2023-11-26T19:01:00, end=2023-11-26T19:03:30) 2023-11-27 13:39:59.556 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=29) - no updates (start=2023-11-26T18:58:30, end=2023-11-26T19:01:00) 2023-11-27 13:39:59.559 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=31) - no updates (start=2023-11-26T19:03:30, end=2023-11-26T19:06:00) 2023-11-27 13:39:59.559 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=32) - no updates (start=2023-11-26T19:06:00, end=2023-11-26T19:08:30) 2023-11-27 13:39:59.560 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=36) - no updates (start=2023-11-26T19:16:00, end=2023-11-26T19:18:30) 2023-11-27 13:39:59.562 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=30) - no updates (start=2023-11-26T19:01:00, end=2023-11-26T19:03:30) 2023-11-27 13:39:59.563 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=37) - no updates (start=2023-11-26T19:18:30, end=2023-11-26T19:21:00) 2023-11-27 13:39:59.564 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=32) - no updates (start=2023-11-26T19:06:00, end=2023-11-26T19:08:30) 2023-11-27 13:39:59.572 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=31) - no updates (start=2023-11-26T19:03:30, end=2023-11-26T19:06:00) 2023-11-27 13:39:59.574 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=33) - no updates (start=2023-11-26T19:08:30, end=2023-11-26T19:11:00) 2023-11-27 13:39:59.576 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=38) - no updates (start=2023-11-26T19:21:00, end=2023-11-26T19:23:30) 2023-11-27 13:39:59.576 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=33) - no updates (start=2023-11-26T19:08:30, end=2023-11-26T19:11:00) 2023-11-27 13:39:59.576 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=32) - no updates (start=2023-11-26T19:06:00, end=2023-11-26T19:08:30) 2023-11-27 13:39:59.586 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=34) - no updates (start=2023-11-26T19:11:00, end=2023-11-26T19:13:30) 2023-11-27 13:39:59.589 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=34) - no updates (start=2023-11-26T19:11:00, end=2023-11-26T19:13:30) 2023-11-27 13:39:59.590 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=33) - no updates (start=2023-11-26T19:08:30, end=2023-11-26T19:11:00) 2023-11-27 13:39:59.592 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=39) - no updates (start=2023-11-26T19:23:30, end=2023-11-26T19:26:00) 2023-11-27 13:39:59.592 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=35) - no updates (start=2023-11-26T19:13:30, end=2023-11-26T19:16:00) 2023-11-27 13:39:59.595 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=34) - no updates (start=2023-11-26T19:11:00, end=2023-11-26T19:13:30) 2023-11-27 13:39:59.598 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=35) - no updates (start=2023-11-26T19:13:30, end=2023-11-26T19:16:00) 2023-11-27 13:39:59.600 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=35) - no updates (start=2023-11-26T19:13:30, end=2023-11-26T19:16:00) 2023-11-27 13:39:59.602 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=36) - no updates (start=2023-11-26T19:16:00, end=2023-11-26T19:18:30) 2023-11-27 13:39:59.604 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=36) - no updates (start=2023-11-26T19:16:00, end=2023-11-26T19:18:30) 2023-11-27 13:39:59.606 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=40) - no updates (start=2023-11-26T19:26:00, end=2023-11-26T19:28:30) 2023-11-27 13:39:59.608 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=36) - no updates (start=2023-11-26T19:16:00, end=2023-11-26T19:18:30) 2023-11-27 13:39:59.612 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=37) - no updates (start=2023-11-26T19:18:30, end=2023-11-26T19:21:00) 2023-11-27 13:39:59.615 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=37) - no updates (start=2023-11-26T19:18:30, end=2023-11-26T19:21:00) 2023-11-27 13:39:59.618 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=41) - no updates (start=2023-11-26T19:28:30, end=2023-11-26T19:31:00) 2023-11-27 13:39:59.618 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=37) - no updates (start=2023-11-26T19:18:30, end=2023-11-26T19:21:00) 2023-11-27 13:39:59.618 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=38) - no updates (start=2023-11-26T19:21:00, end=2023-11-26T19:23:30) 2023-11-27 13:39:59.621 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=38) - no updates (start=2023-11-26T19:21:00, end=2023-11-26T19:23:30) 2023-11-27 13:39:59.623 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=42) - no updates (start=2023-11-26T19:31:00, end=2023-11-26T19:33:30) 2023-11-27 13:39:59.625 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=38) - no updates (start=2023-11-26T19:21:00, end=2023-11-26T19:23:30) 2023-11-27 13:39:59.628 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=39) - no updates (start=2023-11-26T19:23:30, end=2023-11-26T19:26:00) 2023-11-27 13:39:59.629 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=39) - no updates (start=2023-11-26T19:23:30, end=2023-11-26T19:26:00) 2023-11-27 13:39:59.632 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=43) - no updates (start=2023-11-26T19:33:30, end=2023-11-26T19:36:00) 2023-11-27 13:39:59.632 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=39) - no updates (start=2023-11-26T19:23:30, end=2023-11-26T19:26:00) 2023-11-27 13:39:59.645 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=44) - no updates (start=2023-11-26T19:36:00, end=2023-11-26T19:38:30) 2023-11-27 13:39:59.645 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=40) - no updates (start=2023-11-26T19:26:00, end=2023-11-26T19:28:30) 2023-11-27 13:39:59.646 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=40) - no updates (start=2023-11-26T19:26:00, end=2023-11-26T19:28:30) 2023-11-27 13:39:59.646 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=40) - no updates (start=2023-11-26T19:26:00, end=2023-11-26T19:28:30) 2023-11-27 13:39:59.651 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=41) - no updates (start=2023-11-26T19:28:30, end=2023-11-26T19:31:00) 2023-11-27 13:39:59.653 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=45) - no updates (start=2023-11-26T19:38:30, end=2023-11-26T19:41:00) 2023-11-27 13:39:59.655 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=41) - no updates (start=2023-11-26T19:28:30, end=2023-11-26T19:31:00) 2023-11-27 13:39:59.655 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=41) - no updates (start=2023-11-26T19:28:30, end=2023-11-26T19:31:00) 2023-11-27 13:39:59.657 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=42) - no updates (start=2023-11-26T19:31:00, end=2023-11-26T19:33:30) 2023-11-27 13:39:59.662 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=42) - no updates (start=2023-11-26T19:31:00, end=2023-11-26T19:33:30) 2023-11-27 13:39:59.662 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=42) - no updates (start=2023-11-26T19:31:00, end=2023-11-26T19:33:30) 2023-11-27 13:39:59.662 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=46) - no updates (start=2023-11-26T19:41:00, end=2023-11-26T19:43:30) 2023-11-27 13:39:59.664 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=43) - no updates (start=2023-11-26T19:33:30, end=2023-11-26T19:36:00) 2023-11-27 13:39:59.667 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=47) - no updates (start=2023-11-26T19:43:30, end=2023-11-26T19:46:00) 2023-11-27 13:39:59.667 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=43) - no updates (start=2023-11-26T19:33:30, end=2023-11-26T19:36:00) 2023-11-27 13:39:59.667 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=44) - no updates (start=2023-11-26T19:36:00, end=2023-11-26T19:38:30) 2023-11-27 13:39:59.669 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=43) - no updates (start=2023-11-26T19:33:30, end=2023-11-26T19:36:00) 2023-11-27 13:39:59.671 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=45) - no updates (start=2023-11-26T19:38:30, end=2023-11-26T19:41:00) 2023-11-27 13:39:59.671 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=48) - no updates (start=2023-11-26T19:46:00, end=2023-11-26T19:48:30) 2023-11-27 13:39:59.677 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=44) - no updates (start=2023-11-26T19:36:00, end=2023-11-26T19:38:30) 2023-11-27 13:39:59.689 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=44) - no updates (start=2023-11-26T19:36:00, end=2023-11-26T19:38:30) 2023-11-27 13:39:59.689 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=0->1, c=49) - no updates (start=2023-11-26T19:48:30, end=2023-11-26T19:51:00) 2023-11-27 13:39:59.689 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=46) - no updates (start=2023-11-26T19:41:00, end=2023-11-26T19:43:30) 2023-11-27 13:39:59.693 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup delay (cluster=darenOB:10, server=5, level=0->1) - 1066 min 2023-11-27 13:39:59.695 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=47) - no updates (start=2023-11-26T19:43:30, end=2023-11-26T19:46:00) 2023-11-27 13:39:59.696 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=45) - no updates (start=2023-11-26T19:38:30, end=2023-11-26T19:41:00) 2023-11-27 13:39:59.698 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=0) - no updates (start=2023-11-26T17:36:00, end=2023-11-26T17:48:00) 2023-11-27 13:39:59.698 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=45) - no updates (start=2023-11-26T19:38:30, end=2023-11-26T19:41:00) 2023-11-27 13:39:59.703 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=1) - no updates (start=2023-11-26T17:48:00, end=2023-11-26T18:00:00) 2023-11-27 13:39:59.704 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=48) - no updates (start=2023-11-26T19:46:00, end=2023-11-26T19:48:30) 2023-11-27 13:39:59.706 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=46) - no updates (start=2023-11-26T19:41:00, end=2023-11-26T19:43:30) 2023-11-27 13:39:59.706 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=46) - no updates (start=2023-11-26T19:41:00, end=2023-11-26T19:43:30) 2023-11-27 13:39:59.708 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=0->1, c=49) - no updates (start=2023-11-26T19:48:30, end=2023-11-26T19:51:00) 2023-11-27 13:39:59.712 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup delay (cluster=darenOB:10, server=4, level=0->1) - 1066 min 2023-11-27 13:39:59.713 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=47) - no updates (start=2023-11-26T19:43:30, end=2023-11-26T19:46:00) 2023-11-27 13:39:59.713 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=47) - no updates (start=2023-11-26T19:43:30, end=2023-11-26T19:46:00) 2023-11-27 13:39:59.713 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=2) - no updates (start=2023-11-26T18:00:00, end=2023-11-26T18:12:00) 2023-11-27 13:39:59.714 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=0) - no updates (start=2023-11-26T17:36:00, end=2023-11-26T17:48:00) 2023-11-27 13:39:59.717 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=48) - no updates (start=2023-11-26T19:46:00, end=2023-11-26T19:48:30) 2023-11-27 13:39:59.720 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=3) - no updates (start=2023-11-26T18:12:00, end=2023-11-26T18:24:00) 2023-11-27 13:39:59.720 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=48) - no updates (start=2023-11-26T19:46:00, end=2023-11-26T19:48:30) 2023-11-27 13:39:59.722 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=1) - no updates (start=2023-11-26T17:48:00, end=2023-11-26T18:00:00) 2023-11-27 13:39:59.724 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=0->1, c=49) - no updates (start=2023-11-26T19:48:30, end=2023-11-26T19:51:00) 2023-11-27 13:39:59.726 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=0->1, c=49) - no updates (start=2023-11-26T19:48:30, end=2023-11-26T19:51:00) 2023-11-27 13:39:59.726 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=4) - no updates (start=2023-11-26T18:24:00, end=2023-11-26T18:36:00) 2023-11-27 13:39:59.728 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup delay (cluster=darenOB:10, server=6, level=0->1) - 1066 min 2023-11-27 13:39:59.729 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup delay (cluster=darenOB:10, server=3, level=0->1) - 1066 min 2023-11-27 13:39:59.732 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=0) - no updates (start=2023-11-26T17:36:00, end=2023-11-26T17:48:00) 2023-11-27 13:39:59.732 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=0) - no updates (start=2023-11-26T17:36:00, end=2023-11-26T17:48:00) 2023-11-27 13:39:59.732 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=5) - no updates (start=2023-11-26T18:36:00, end=2023-11-26T18:48:00) 2023-11-27 13:39:59.732 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=2) - no updates (start=2023-11-26T18:00:00, end=2023-11-26T18:12:00) 2023-11-27 13:39:59.737 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=1) - no updates (start=2023-11-26T17:48:00, end=2023-11-26T18:00:00) 2023-11-27 13:39:59.739 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=1) - no updates (start=2023-11-26T17:48:00, end=2023-11-26T18:00:00) 2023-11-27 13:39:59.739 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=6) - no updates (start=2023-11-26T18:48:00, end=2023-11-26T19:00:00) 2023-11-27 13:39:59.739 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=3) - no updates (start=2023-11-26T18:12:00, end=2023-11-26T18:24:00) 2023-11-27 13:39:59.745 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=2) - no updates (start=2023-11-26T18:00:00, end=2023-11-26T18:12:00) 2023-11-27 13:39:59.746 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=2) - no updates (start=2023-11-26T18:00:00, end=2023-11-26T18:12:00) 2023-11-27 13:39:59.747 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=7) - no updates (start=2023-11-26T19:00:00, end=2023-11-26T19:12:00) 2023-11-27 13:39:59.750 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=4) - no updates (start=2023-11-26T18:24:00, end=2023-11-26T18:36:00) 2023-11-27 13:39:59.751 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=3) - no updates (start=2023-11-26T18:12:00, end=2023-11-26T18:24:00) 2023-11-27 13:39:59.751 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=3) - no updates (start=2023-11-26T18:12:00, end=2023-11-26T18:24:00) 2023-11-27 13:39:59.754 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=8) - no updates (start=2023-11-26T19:12:00, end=2023-11-26T19:24:00) 2023-11-27 13:39:59.757 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=4) - no updates (start=2023-11-26T18:24:00, end=2023-11-26T18:36:00) 2023-11-27 13:39:59.758 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=5) - no updates (start=2023-11-26T18:36:00, end=2023-11-26T18:48:00) 2023-11-27 13:39:59.758 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=4) - no updates (start=2023-11-26T18:24:00, end=2023-11-26T18:36:00) 2023-11-27 13:39:59.761 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=9) - no updates (start=2023-11-26T19:24:00, end=2023-11-26T19:36:00) 2023-11-27 13:39:59.763 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=6) - no updates (start=2023-11-26T18:48:00, end=2023-11-26T19:00:00) 2023-11-27 13:39:59.763 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=5) - no updates (start=2023-11-26T18:36:00, end=2023-11-26T18:48:00) 2023-11-27 13:39:59.763 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=5) - no updates (start=2023-11-26T18:36:00, end=2023-11-26T18:48:00) 2023-11-27 13:39:59.766 WARN 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=5, level=1->2, c=10) - no updates (start=2023-11-26T19:36:00, end=2023-11-26T19:48:00) 2023-11-27 13:39:59.768 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=6) - no updates (start=2023-11-26T18:48:00, end=2023-11-26T19:00:00) 2023-11-27 13:39:59.768 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=7) - no updates (start=2023-11-26T19:00:00, end=2023-11-26T19:12:00) 2023-11-27 13:39:59.769 INFO 10 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=darenOB:10, server=5, level=1->2, c=11) - not enough cached interval (source=2023-11-26T19:51:00, target=2023-11-26T19:48:00, granule=600000ms, delay=180000ms) 2023-11-27 13:39:59.769 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=6) - no updates (start=2023-11-26T18:48:00, end=2023-11-26T19:00:00) 2023-11-27 13:39:59.773 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=7) - no updates (start=2023-11-26T19:00:00, end=2023-11-26T19:12:00) 2023-11-27 13:39:59.773 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=7) - no updates (start=2023-11-26T19:00:00, end=2023-11-26T19:12:00) 2023-11-27 13:39:59.773 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=8) - no updates (start=2023-11-26T19:12:00, end=2023-11-26T19:24:00) 2023-11-27 13:39:59.777 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=9) - no updates (start=2023-11-26T19:24:00, end=2023-11-26T19:36:00) 2023-11-27 13:39:59.777 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=8) - no updates (start=2023-11-26T19:12:00, end=2023-11-26T19:24:00) 2023-11-27 13:39:59.777 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=8) - no updates (start=2023-11-26T19:12:00, end=2023-11-26T19:24:00) 2023-11-27 13:39:59.785 WARN 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=4, level=1->2, c=10) - no updates (start=2023-11-26T19:36:00, end=2023-11-26T19:48:00) 2023-11-27 13:39:59.786 INFO 10 --- [sql-rollup-worker-4,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=darenOB:10, server=4, level=1->2, c=11) - not enough cached interval (source=2023-11-26T19:51:00, target=2023-11-26T19:48:00, granule=600000ms, delay=180000ms) 2023-11-27 13:39:59.794 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=9) - no updates (start=2023-11-26T19:24:00, end=2023-11-26T19:36:00) 2023-11-27 13:39:59.797 WARN 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=3, level=1->2, c=10) - no updates (start=2023-11-26T19:36:00, end=2023-11-26T19:48:00) 2023-11-27 13:39:59.798 INFO 10 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=darenOB:10, server=3, level=1->2, c=11) - not enough cached interval (source=2023-11-26T19:51:00, target=2023-11-26T19:48:00, granule=600000ms, delay=180000ms) 2023-11-27 13:39:59.804 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=9) - no updates (start=2023-11-26T19:24:00, end=2023-11-26T19:36:00) 2023-11-27 13:39:59.819 WARN 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=darenOB:10, server=6, level=1->2, c=10) - no updates (start=2023-11-26T19:36:00, end=2023-11-26T19:48:00) 2023-11-27 13:39:59.820 INFO 10 --- [sql-rollup-worker-5,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=darenOB:10, server=6, level=1->2, c=11) - not enough cached interval (source=2023-11-26T19:51:00, target=2023-11-26T19:48:00, granule=600000ms, delay=180000ms) 2023-11-27 13:39:59.820 INFO 10 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=913ms, period=120000ms 2023-11-27 13:40:00.000 INFO 10 --- [pool-ocp-schedules-10,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-11-27 13:40:00.000 INFO 10 --- [pool-ocp-schedules-10,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1701063600 2023-11-27 13:40:00.000 INFO 10 --- [pool-ocp-schedules-9,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-11-27 13:40:00.000 INFO 10 --- [pool-ocp-schedules-9,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-11-27 13:40:00.038 INFO 10 --- [pool-ocp-schedules-9,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-11-27 13:40:00.106 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063600, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:00.106 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063600, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:00.106 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063600, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:00.106 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063600, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:00.106 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063600, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:00.687 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:36:07.0, nextRun=2023-11-27 13:40:00.0, currentDate=Mon Nov 27 13:40:00 CST 2023 2023-11-27 13:40:00.687 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=12, name=Refresh agent config, templateName=RefreshAgentConfigSchedule, scheduleRule=0 */10 * * * *, lastRunTime=2023-11-27 13:36:07.0, nextRunTime=2023-11-27 13:40:00.0} 2023-11-27 13:40:00.718 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Refresh agent config, nodeCount=1, taskCount=1 2023-11-27 13:40:00.718 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:40:00.718 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Refresh agent config, taskDefinitionId=12, state=RUNNING, operation=EXECUTE} 2023-11-27 13:40:00.739 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:38:00.0, nextRun=2023-11-27 13:40:00.0, currentDate=Mon Nov 27 13:40:00 CST 2023 2023-11-27 13:40:00.739 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=14, name=Sync root service list, templateName=SyncRsListSchedule, scheduleRule=0 */2 * * * ?, lastRunTime=2023-11-27 13:38:00.0, nextRunTime=2023-11-27 13:40:00.0} 2023-11-27 13:40:00.749 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync root service list, nodeCount=1, taskCount=2 2023-11-27 13:40:00.749 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:40:00.749 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:40:00.749 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync root service list, taskDefinitionId=14, state=RUNNING, operation=EXECUTE} 2023-11-27 13:40:00.757 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:36:08.0, nextRun=2023-11-27 13:40:00.0, currentDate=Mon Nov 27 13:40:00 CST 2023 2023-11-27 13:40:00.757 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=15, name=Sync system event, templateName=SyncSystemEventSchedule, scheduleRule=0 */10 * * * *, lastRunTime=2023-11-27 13:36:08.0, nextRunTime=2023-11-27 13:40:00.0} 2023-11-27 13:40:00.763 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync system event, nodeCount=2, taskCount=3 2023-11-27 13:40:00.763 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:40:00.763 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync system event, taskDefinitionId=15, state=RUNNING, operation=EXECUTE} 2023-11-27 13:40:00.777 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:36:09.0, nextRun=2023-11-27 13:40:00.0, currentDate=Mon Nov 27 13:40:00 CST 2023 2023-11-27 13:40:00.777 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=20, name=Check all cluster configUrl, templateName=CheckClusterConfigUrlSchedule, scheduleRule=0 */5 * * * ?, lastRunTime=2023-11-27 13:36:09.0, nextRunTime=2023-11-27 13:40:00.0} 2023-11-27 13:40:00.781 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=CheckClusterConfigUrlSchedule, nodeCount=1, taskCount=1 2023-11-27 13:40:00.781 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:40:00.781 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=CheckClusterConfigUrlSchedule, taskDefinitionId=20, state=RUNNING, operation=EXECUTE} 2023-11-27 13:40:00.790 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:36:09.0, nextRun=2023-11-27 13:40:00.0, currentDate=Mon Nov 27 13:40:00 CST 2023 2023-11-27 13:40:00.790 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=21, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2023-11-27 13:36:09.0, nextRunTime=2023-11-27 13:40:00.0} 2023-11-27 13:40:00.861 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2023-11-27 13:40:00.861 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:40:00.861 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=SqlParseTaskSchedule, taskDefinitionId=21, state=RUNNING, operation=EXECUTE} 2023-11-27 13:40:00.868 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:39:01.0, nextRun=2023-11-27 13:40:00.0, currentDate=Mon Nov 27 13:40:00 CST 2023 2023-11-27 13:40:00.868 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-27 13:39:01.0, nextRunTime=2023-11-27 13:40:00.0} 2023-11-27 13:40:00.872 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-11-27 13:40:00.872 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:40:00.872 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-11-27 13:40:00.877 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:39:01.0, nextRun=2023-11-27 13:40:00.0, currentDate=Mon Nov 27 13:40:00 CST 2023 2023-11-27 13:40:00.878 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-27 13:39:01.0, nextRunTime=2023-11-27 13:40:00.0} 2023-11-27 13:40:00.888 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-11-27 13:40:00.888 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:40:00.888 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-11-27 13:40:00.898 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:39:01.0, nextRun=2023-11-27 13:40:00.0, currentDate=Mon Nov 27 13:40:00 CST 2023 2023-11-27 13:40:00.898 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-27 13:39:01.0, nextRunTime=2023-11-27 13:40:00.0} 2023-11-27 13:40:00.912 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-11-27 13:40:00.912 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:40:00.913 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-11-27 13:40:00.926 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:39:31.0, nextRun=2023-11-27 13:40:00.0, currentDate=Mon Nov 27 13:40:00 CST 2023 2023-11-27 13:40:00.926 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-11-27 13:39:31.0, nextRunTime=2023-11-27 13:40:00.0} 2023-11-27 13:40:00.931 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-11-27 13:40:00.931 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:40:00.932 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-11-27 13:40:00.937 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:36:11.0, nextRun=2023-11-27 13:40:00.0, currentDate=Mon Nov 27 13:40:00 CST 2023 2023-11-27 13:40:00.938 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=35, name=Collect all tenants mini compactions, templateName=CollectTenantMiniCompactionSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2023-11-27 13:36:11.0, nextRunTime=2023-11-27 13:40:00.0} 2023-11-27 13:40:00.941 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000072, operation:EXECUTE, state: RUNNING 2023-11-27 13:40:00.943 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants mini compactions, nodeCount=1, taskCount=1 2023-11-27 13:40:00.943 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:40:00.943 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants mini compactions, taskDefinitionId=35, state=RUNNING, operation=EXECUTE} 2023-11-27 13:40:00.944 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000072, name=Refresh all agent config, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.RefreshAllAgentConfigTask, seriesId=0, startTime=2023-11-27T13:40:00.937+08:00, endTime=null} 2023-11-27 13:40:00.950 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:36:11.0, nextRun=2023-11-27 13:40:00.0, currentDate=Mon Nov 27 13:40:00 CST 2023 2023-11-27 13:40:00.950 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=37, name=HighRiskSqlSchedule, templateName=HighRiskSqlSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2023-11-27 13:36:11.0, nextRunTime=2023-11-27 13:40:00.0} 2023-11-27 13:40:00.965 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000073, operation:EXECUTE, state: RUNNING 2023-11-27 13:40:00.969 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000073, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=1, startTime=2023-11-27T13:40:00.962+08:00, endTime=null} 2023-11-27 13:40:00.970 INFO 10 --- [pool-prometheus-minute-schedule-2,,66c4fd04e9a1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1701063660, interval=60 2023-11-27 13:40:00.973 INFO 10 --- [pool-prometheus-minute-schedule-1,,fc1987a286a4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1701063660, interval=60 2023-11-27 13:40:00.995 INFO 10 --- [pool-prometheus-minute-schedule-2,,66c4fd04e9a1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=11, collectTaskCount=11 2023-11-27 13:40:00.995 INFO 10 --- [pool-prometheus-minute-schedule-2,,66c4fd04e9a1] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1701063600, count=11, elapsed=1699362537395 2023-11-27 13:40:00.996 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/extra, collectAt=1701063600, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:00.996 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62888/metrics/stat, collectAt=1701063600, message=java.net.ConnectException: Connection refused: /133.197.206.97:62888, rootCause=ConnectException: Connection refused 2023-11-27 13:40:00.996 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/stat, collectAt=1701063600, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:00.998 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62888/metrics/stat, collectAt=1701063600, message=java.net.ConnectException: Connection refused: /133.197.206.88:62888, rootCause=ConnectException: Connection refused 2023-11-27 13:40:01.002 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,fad3c2ef2505] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000072, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000061, task_operation=execute, latest_execution_start_time=2023-11-27T13:40:00.937+08:00, sub_task_instance_id=10000072}, listMap={}}, executor=10.120.0.1 2023-11-27 13:40:01.005 INFO 10 --- [pool-prometheus-minute-schedule-1,,fc1987a286a4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2023-11-27 13:40:01.005 INFO 10 --- [pool-prometheus-minute-schedule-1,,fc1987a286a4] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1701063600, count=8, elapsed=1699362537405 2023-11-27 13:40:01.006 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000074, operation:EXECUTE, state: RUNNING 2023-11-27 13:40:01.010 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,d785b9bae35e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000073, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=8000062, task_operation=execute, latest_execution_start_time=2023-11-27T13:40:00.961+08:00, sub_task_instance_id=10000073}, listMap={cluster_ids=[1, 2]}}, executor=10.120.0.1 2023-11-27 13:40:01.011 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/stat, collectAt=1701063600, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:01.011 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/stat, collectAt=1701063600, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:01.011 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62888/metrics/stat, collectAt=1701063600, message=java.net.ConnectException: Connection refused: /133.197.206.92:62888, rootCause=ConnectException: Connection refused 2023-11-27 13:40:01.046 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000074, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=0, startTime=2023-11-27T13:40:00.993+08:00, endTime=null} 2023-11-27 13:40:01.046 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,fad3c2ef2505] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=6 2023-11-27 13:40:01.047 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=HighRiskSqlAuditStatTaskSchedule, nodeCount=1, taskCount=1 2023-11-27 13:40:01.047 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:40:01.047 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=HighRiskSqlAuditStatTaskSchedule, taskDefinitionId=37, state=RUNNING, operation=EXECUTE} 2023-11-27 13:40:01.047 INFO 10 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=5 2023-11-27 13:40:01.047 INFO 10 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=4 2023-11-27 13:40:01.050 INFO 10 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=7 2023-11-27 13:40:01.066 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:39:01.0, nextRun=2023-11-27 13:40:00.0, currentDate=Mon Nov 27 13:40:01 CST 2023 2023-11-27 13:40:01.066 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=1000002, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-27 13:39:01.0, nextRunTime=2023-11-27 13:40:00.0} 2023-11-27 13:40:01.070 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-11-27 13:40:01.070 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:40:01.070 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=1000002, state=RUNNING, operation=EXECUTE} 2023-11-27 13:40:01.117 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000075, operation:EXECUTE, state: RUNNING 2023-11-27 13:40:01.143 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000075, name=Wait node, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=0, startTime=2023-11-27T13:40:01.087+08:00, endTime=null} 2023-11-27 13:40:01.182 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:40:01.189 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:40:01.196 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000078, operation:EXECUTE, state: RUNNING 2023-11-27 13:40:01.211 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000078, name=Check root service , state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CheckClusterRootServersTask, seriesId=0, startTime=2023-11-27T13:40:01.190+08:00, endTime=null} 2023-11-27 13:40:01.219 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,d785b9bae35e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:01.220 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,d785b9bae35e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:01.220 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,d785b9bae35e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:01.221 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,d785b9bae35e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:01.223 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,4a48ddf9ad11] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000074, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=8000062, task_operation=execute, latest_execution_start_time=2023-11-27T13:40:00.990+08:00, sub_task_instance_id=10000074}, listMap={cluster_ids=[1, 2]}}, executor=10.120.0.1 2023-11-27 13:40:01.230 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,d785b9bae35e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:01.231 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,d785b9bae35e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:01.234 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,d785b9bae35e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2023-11-27 13:40:01.254 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,4a48ddf9ad11] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000074, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:40:01.258 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,d785b9bae35e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:01.259 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,d785b9bae35e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:01.260 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,d785b9bae35e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [133.197.204.10, 2882] 2023-11-27 13:40:01.264 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,d785b9bae35e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:01.264 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,d785b9bae35e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:01.265 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,d785b9bae35e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [133.197.204.11, 2882] 2023-11-27 13:40:01.267 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,d785b9bae35e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:01.267 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,d785b9bae35e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:01.269 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,d785b9bae35e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [133.197.204.12, 2882] 2023-11-27 13:40:01.269 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000079, operation:EXECUTE, state: RUNNING 2023-11-27 13:40:01.271 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,d785b9bae35e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:01.271 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,d785b9bae35e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:01.272 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,d785b9bae35e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [133.197.204.13, 2882] 2023-11-27 13:40:01.273 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,d785b9bae35e] c.o.o.s.t.b.cluster.UpdateRsListTask : assembled rootserver_json={"ObCluster":"darenOB","ObClusterId":10,"ObRegion":"darenOB","ObRegionId":10,"ReadonlyRsList":[],"RsList":[{"address":"133.197.204.10:2882","role":"LEADER","sql_port":2881},{"address":"133.197.204.11:2882","role":"FOLLOWER","sql_port":2881},{"address":"133.197.204.12:2882","role":"FOLLOWER","sql_port":2881},{"address":"133.197.204.13:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1701063601273000} 2023-11-27 13:40:01.278 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000079, name=Sql Parse Task, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=2023-11-27T13:40:01.264+08:00, endTime=null} 2023-11-27 13:40:01.284 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,d785b9bae35e] c.o.o.s.t.b.cluster.UpdateRsListTask : update rootserver_json success, clusterName=darenOB, obClusterId=10 2023-11-27 13:40:01.288 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,4f1f84f47bf5] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000075, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000063, task_operation=execute, latest_execution_start_time=2023-11-27T13:40:01.082+08:00, sub_task_instance_id=10000075}, listMap={cluster_ids=[1, 2]}}, executor=10.120.0.1 2023-11-27 13:40:01.311 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,d785b9bae35e] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000073, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:40:01.315 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,85d74fb8d546] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000078, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000064, task_operation=execute, latest_execution_start_time=2023-11-27T13:40:01.190+08:00, sub_task_instance_id=10000078}, listMap={}}, executor=10.120.0.1 2023-11-27 13:40:01.319 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,85d74fb8d546] c.o.o.s.t.b.c.AbstractAllClusterTask : Check root service executed completed, tryCount=2, successCount=0 2023-11-27 13:40:01.330 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,4f1f84f47bf5] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000075, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:40:01.358 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,85d74fb8d546] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000078, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:40:01.425 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,fad3c2ef2505] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 6 agent sql collect property 2023-11-27 13:40:01.436 INFO 10 --- [ForkJoinPool.commonPool-worker-3,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 7 agent sql collect property 2023-11-27 13:40:01.438 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000080, operation:EXECUTE, state: RUNNING 2023-11-27 13:40:01.447 INFO 10 --- [ForkJoinPool.commonPool-worker-1,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 4 agent sql collect property 2023-11-27 13:40:01.451 INFO 10 --- [ForkJoinPool.commonPool-worker-0,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 5 agent sql collect property 2023-11-27 13:40:01.469 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000080, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-11-27T13:40:01.397+08:00, endTime=null} 2023-11-27 13:40:01.477 INFO 10 --- [ForkJoinPool-41-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:01.479 INFO 10 --- [ForkJoinPool-41-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:01.479 INFO 10 --- [ForkJoinPool-41-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:01.479 INFO 10 --- [ForkJoinPool-41-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:01.481 INFO 10 --- [ForkJoinPool-41-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:01.482 INFO 10 --- [ForkJoinPool-41-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:01.482 INFO 10 --- [ForkJoinPool-41-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:01.486 INFO 10 --- [ForkJoinPool-41-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:01.486 INFO 10 --- [ForkJoinPool-41-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:01.486 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b7ed28eb4341] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000079, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=8000065, task_operation=execute, latest_execution_start_time=2023-11-27T13:40:01.263+08:00, sub_task_instance_id=10000079}, listMap={cluster_ids=[[1,1,2]], tenant_ids=[[3,2,5]]}}, executor=10.120.0.1 2023-11-27 13:40:01.486 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b7ed28eb4341] c.o.o.p.sql.internal.task.SqlParseTask : Parse sql for [3, 2, 5] 2023-11-27 13:40:01.486 INFO 10 --- [ForkJoinPool-41-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:01.487 INFO 10 --- [ForkJoinPool-41-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2023-11-27 13:40:01.489 INFO 10 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-11-27 13:40:01.491 INFO 10 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-11-27 13:40:01.492 INFO 10 --- [ForkJoinPool-41-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:01.493 INFO 10 --- [ForkJoinPool-41-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:01.494 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,fad3c2ef2505] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2023-11-27 13:40:01.495 INFO 10 --- [ForkJoinPool-41-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2023-11-27 13:40:01.496 INFO 10 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2023-11-27 13:40:01.512 INFO 10 --- [http-nio-0.0.0.0-8080-exec-6,e06cdf3f33f3438f,5dcffe71917f] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=obcluster&ObClusterId=1&version=2, client=133.197.206.97, traceId=e06cdf3f33f3438f, method=OcpApiRtnObj com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obcluster,1,false,2,] 2023-11-27 13:40:01.515 INFO 10 --- [http-nio-0.0.0.0-8080-exec-6,e06cdf3f33f3438f,5dcffe71917f] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.206.97, traceId=e06cdf3f33f3438f, duration=3 ms] 2023-11-27 13:40:01.519 INFO 10 --- [http-nio-0.0.0.0-8080-exec-9,e9a9d0bb5a294da8,f3a45394546d] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=darenOB&ObClusterId=10&version=2, client=133.197.206.97, traceId=e9a9d0bb5a294da8, method=OcpApiRtnObj com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=darenOB,10,false,2,] 2023-11-27 13:40:01.520 INFO 10 --- [ForkJoinPool-41-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:01.520 INFO 10 --- [ForkJoinPool-41-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:01.521 INFO 10 --- [ForkJoinPool-41-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2023-11-27 13:40:01.522 INFO 10 --- [http-nio-0.0.0.0-8080-exec-9,e9a9d0bb5a294da8,f3a45394546d] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.206.97, traceId=e9a9d0bb5a294da8, duration=3 ms] 2023-11-27 13:40:01.529 INFO 10 --- [ForkJoinPool-41-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:01.529 INFO 10 --- [ForkJoinPool-41-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:01.530 INFO 10 --- [ForkJoinPool-41-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:01.530 INFO 10 --- [ForkJoinPool-41-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:01.531 INFO 10 --- [ForkJoinPool-41-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2023-11-27 13:40:01.536 INFO 10 --- [ForkJoinPool-41-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, stop_time, start_service_time, block_migrate_in_time FROM oceanbase.__all_server where svr_ip = ? and svr_port = ?, args: [133.197.206.97, 2882] 2023-11-27 13:40:01.544 INFO 10 --- [ForkJoinPool-41-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:01.544 INFO 10 --- [ForkJoinPool-41-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:01.545 INFO 10 --- [ForkJoinPool-41-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [133.197.204.10, 2882] 2023-11-27 13:40:01.549 INFO 10 --- [ForkJoinPool-41-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:01.549 INFO 10 --- [ForkJoinPool-41-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:01.550 INFO 10 --- [ForkJoinPool-41-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [133.197.204.11, 2882] 2023-11-27 13:40:01.554 INFO 10 --- [ForkJoinPool-41-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:01.554 INFO 10 --- [ForkJoinPool-41-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:01.555 INFO 10 --- [ForkJoinPool-41-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [133.197.204.12, 2882] 2023-11-27 13:40:01.557 INFO 10 --- [ForkJoinPool-41-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:01.557 INFO 10 --- [ForkJoinPool-41-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:01.561 INFO 10 --- [ForkJoinPool-41-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [133.197.204.13, 2882] 2023-11-27 13:40:01.611 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,5a8e65b49184] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000080, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000066, task_operation=execute, latest_execution_start_time=2023-11-27T13:40:01.385+08:00, sub_task_instance_id=10000080}, listMap={}}, executor=10.120.0.1 2023-11-27 13:40:01.650 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=6 2023-11-27 13:40:01.652 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,fad3c2ef2505] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.12:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(monagent.ob.monitor.password=xxx (monagent.host.ip=133.197.204.12), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.ob.cluster.name=darenOB), (monagent.ob.cluster.id=10), (monagent.cluster.id=2), (monagent.ob.zone.name=zone5), (monagent.ob.sql.port=2881), (monagent.ob.rpc.port=2882), (ob.log.path=/home/obdata/log), (ob.data.path=/home/obdata/data), (monagent.pipeline.ob.status=active), (ob.logcleaner.enabled=true), (ob.logtailer.enabled=true), (ob.install.path=/home/obdata/admin/oceanbase), (monagent.log.level=info), (mgragent.log.level=info), (monagent.monitordb.host=133.197.206.97), (monagent.monitordb.port=2883), (monagent.monitordb.user=monitor_user@monitor_tenant#obcluster), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.obproxy.cluster.id=4), (monagent.obproxy.cluster.name=darenOBProxy), (monagent.host.ip=133.197.204.12), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.log.level=info), (mgragent.log.level=info), (monagent.obproxy.sql.port=2883), (monagent.pipeline.obproxy.status=active), (monagent.obproxy.exporter.port=2884), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["is_out_of_memstore_mem=true","right_to_die_or_duty_to_live","on_fatal_error"],"level":1,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["partition table update task cost too much time to execute","clog disk is almost full","set_disk_error:attention!!!","error occurs on data disk or slog disk"],"level":2,"logLevels":["ERROR","WARNING"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"election_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"election_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_all_error_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_expanse_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_expanse_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"obproxy_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"obproxy_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_sql_execute_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"sql_execute_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_compress_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_compress_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"message_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["kernel: nvme"],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"rootservice_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"rootservice_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_sql_query_slow","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"slow_sql_query_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300}]), (monagent.pipeline.sql.audit.status=active), (monagent.pipeline.sql.plan.status=active), (monagent.pipeline.plan.monitor.status=active), (monagent.pipeline.transaction.status=active), (monagent.pipeline.session.status=active), (monagent.pipeline.slow.sql.status=active)]), params:null 2023-11-27 13:40:01.654 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=9 2023-11-27 13:40:01.654 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-11-27 13:40:01.654 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2023-11-27 13:40:01.655 INFO 10 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.10:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(monagent.ob.monitor.password=xxx (monagent.host.ip=133.197.204.10), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.ob.cluster.name=darenOB), (monagent.ob.cluster.id=10), (monagent.cluster.id=2), (monagent.ob.zone.name=zone3), (monagent.ob.sql.port=2881), (monagent.ob.rpc.port=2882), (ob.log.path=/home/obdata/log), (ob.data.path=/home/obdata/data), (monagent.pipeline.ob.status=active), (ob.logcleaner.enabled=true), (ob.logtailer.enabled=true), (ob.install.path=/home/obdata/admin/oceanbase), (monagent.log.level=info), (mgragent.log.level=info), (monagent.monitordb.host=133.197.206.97), (monagent.monitordb.port=2883), (monagent.monitordb.user=monitor_user@monitor_tenant#obcluster), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.obproxy.cluster.id=4), (monagent.obproxy.cluster.name=darenOBProxy), (monagent.host.ip=133.197.204.10), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.log.level=info), (mgragent.log.level=info), (monagent.obproxy.sql.port=2883), (monagent.pipeline.obproxy.status=active), (monagent.obproxy.exporter.port=2884), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["is_out_of_memstore_mem=true","right_to_die_or_duty_to_live","on_fatal_error"],"level":1,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["partition table update task cost too much time to execute","clog disk is almost full","set_disk_error:attention!!!","error occurs on data disk or slog disk"],"level":2,"logLevels":["ERROR","WARNING"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"election_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"election_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_all_error_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_expanse_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_expanse_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"obproxy_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"obproxy_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_sql_execute_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"sql_execute_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_compress_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_compress_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"message_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["kernel: nvme"],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"rootservice_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"rootservice_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_sql_query_slow","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"slow_sql_query_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300}]), (monagent.pipeline.sql.audit.status=active), (monagent.pipeline.sql.plan.status=active), (monagent.pipeline.plan.monitor.status=active), (monagent.pipeline.transaction.status=active), (monagent.pipeline.session.status=active), (monagent.pipeline.slow.sql.status=active)]), params:null 2023-11-27 13:40:01.670 INFO 10 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 4, do nothing 2023-11-27 13:40:01.672 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2023-11-27 13:40:01.672 INFO 10 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.13:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(monagent.ob.monitor.password=xxx (monagent.host.ip=133.197.204.13), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.ob.cluster.name=darenOB), (monagent.ob.cluster.id=10), (monagent.cluster.id=2), (monagent.ob.zone.name=zone6), (monagent.ob.sql.port=2881), (monagent.ob.rpc.port=2882), (ob.log.path=/home/obdata/log), (ob.data.path=/home/obdata/data), (monagent.pipeline.ob.status=active), (ob.logcleaner.enabled=true), (ob.logtailer.enabled=true), (ob.install.path=/home/obdata/admin/oceanbase), (monagent.log.level=info), (mgragent.log.level=info), (monagent.monitordb.host=133.197.206.97), (monagent.monitordb.port=2883), (monagent.monitordb.user=monitor_user@monitor_tenant#obcluster), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.obproxy.cluster.id=4), (monagent.obproxy.cluster.name=darenOBProxy), (monagent.host.ip=133.197.204.13), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.log.level=info), (mgragent.log.level=info), (monagent.obproxy.sql.port=2883), (monagent.pipeline.obproxy.status=active), (monagent.obproxy.exporter.port=2884), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["is_out_of_memstore_mem=true","right_to_die_or_duty_to_live","on_fatal_error"],"level":1,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["partition table update task cost too much time to execute","clog disk is almost full","set_disk_error:attention!!!","error occurs on data disk or slog disk"],"level":2,"logLevels":["ERROR","WARNING"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"election_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"election_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_all_error_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_expanse_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_expanse_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"obproxy_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"obproxy_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_sql_execute_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"sql_execute_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_compress_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_compress_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"message_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["kernel: nvme"],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"rootservice_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"rootservice_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_sql_query_slow","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"slow_sql_query_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300}]), (monagent.pipeline.sql.audit.status=active), (monagent.pipeline.sql.plan.status=active), (monagent.pipeline.plan.monitor.status=active), (monagent.pipeline.transaction.status=active), (monagent.pipeline.session.status=active), (monagent.pipeline.slow.sql.status=active)]), params:null 2023-11-27 13:40:01.679 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,fad3c2ef2505] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 6, do nothing 2023-11-27 13:40:01.681 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:40:01.683 INFO 10 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.11:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(monagent.ob.monitor.password=xxx (monagent.host.ip=133.197.204.11), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.ob.cluster.name=darenOB), (monagent.ob.cluster.id=10), (monagent.cluster.id=2), (monagent.ob.zone.name=zone4), (monagent.ob.sql.port=2881), (monagent.ob.rpc.port=2882), (ob.log.path=/home/obdata/log), (ob.data.path=/home/obdata/data), (monagent.pipeline.ob.status=active), (ob.logcleaner.enabled=true), (ob.logtailer.enabled=true), (ob.install.path=/home/obdata/admin/oceanbase), (monagent.log.level=info), (mgragent.log.level=info), (monagent.monitordb.host=133.197.206.97), (monagent.monitordb.port=2883), (monagent.monitordb.user=monitor_user@monitor_tenant#obcluster), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.obproxy.cluster.id=4), (monagent.obproxy.cluster.name=darenOBProxy), (monagent.host.ip=133.197.204.11), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.log.level=info), (mgragent.log.level=info), (monagent.obproxy.sql.port=2883), (monagent.pipeline.obproxy.status=active), (monagent.obproxy.exporter.port=2884), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["is_out_of_memstore_mem=true","right_to_die_or_duty_to_live","on_fatal_error"],"level":1,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["partition table update task cost too much time to execute","clog disk is almost full","set_disk_error:attention!!!","error occurs on data disk or slog disk"],"level":2,"logLevels":["ERROR","WARNING"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"election_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"election_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_all_error_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_expanse_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_expanse_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"obproxy_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"obproxy_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_sql_execute_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"sql_execute_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_compress_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_compress_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"message_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["kernel: nvme"],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"rootservice_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"rootservice_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_sql_query_slow","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"slow_sql_query_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300}]), (monagent.pipeline.sql.audit.status=active), (monagent.pipeline.sql.plan.status=active), (monagent.pipeline.plan.monitor.status=active), (monagent.pipeline.transaction.status=active), (monagent.pipeline.session.status=active), (monagent.pipeline.slow.sql.status=active)]), params:null 2023-11-27 13:40:01.684 INFO 10 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 7, do nothing 2023-11-27 13:40:01.686 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,5a8e65b49184] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000080, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:40:01.694 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2023-11-27 13:40:01.699 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:40:01.699 INFO 10 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 5, do nothing 2023-11-27 13:40:01.735 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,fad3c2ef2505] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000072, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:40:01.744 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.12:62889/api/v1/log/alarms, params:null 2023-11-27 13:40:01.747 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-11-27 13:40:01.751 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.92:62889/api/v1/log/alarms, params:null 2023-11-27 13:40:01.757 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-11-27 13:40:01.776 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:40:01.784 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-11-27 13:40:01.807 INFO 10 --- [pool-overview-container-1,,f7f3a50920f6] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=417, alarmHistory size=15889 2023-11-27 13:40:01.807 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.13:62889/api/v1/log/alarms, params:null 2023-11-27 13:40:01.818 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.88:62889/api/v1/log/alarms, params:null 2023-11-27 13:40:01.822 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-11-27 13:40:01.834 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:40:01.860 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.3:62889/api/v1/log/alarms, params:null 2023-11-27 13:40:01.860 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.11:62889/api/v1/log/alarms, params:null 2023-11-27 13:40:01.879 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.97:62889/api/v1/log/alarms, params:null 2023-11-27 13:40:01.948 INFO 10 --- [pool-scanner--12415-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=76 2023-11-27 13:40:02.615 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000081, operation:EXECUTE, state: RUNNING 2023-11-27 13:40:02.654 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000081, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-11-27T13:40:02.582+08:00, endTime=null} 2023-11-27 13:40:02.762 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000082, operation:EXECUTE, state: RUNNING 2023-11-27 13:40:02.769 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000082, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-11-27T13:40:02.747+08:00, endTime=null} 2023-11-27 13:40:02.774 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000061, status:SUCCESS 2023-11-27 13:40:02.775 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000061, status:SUCCESS 2023-11-27 13:40:02.797 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000083, operation:EXECUTE, state: RUNNING 2023-11-27 13:40:02.832 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000083, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-11-27T13:40:02.793+08:00, endTime=null} 2023-11-27 13:40:02.886 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,85e02cdbd805] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000082, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000068, task_operation=execute, latest_execution_start_time=2023-11-27T13:40:02.744+08:00, sub_task_instance_id=10000082}, listMap={}}, executor=10.120.0.1 2023-11-27 13:40:02.927 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000062, status:SUCCESS 2023-11-27 13:40:02.933 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000062, status:SUCCESS 2023-11-27 13:40:03.042 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,ba9ea31eae89] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000081, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000067, task_operation=execute, latest_execution_start_time=2023-11-27T13:40:02.576+08:00, sub_task_instance_id=10000081}, listMap={}}, executor=10.120.0.1 2023-11-27 13:40:03.113 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000084, operation:EXECUTE, state: RUNNING 2023-11-27 13:40:03.179 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000064, status:SUCCESS 2023-11-27 13:40:03.182 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000084, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask, seriesId=0, startTime=2023-11-27T13:40:03.075+08:00, endTime=null} 2023-11-27 13:40:03.184 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,2fa4e1bf579d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000083, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000069, task_operation=execute, latest_execution_start_time=2023-11-27T13:40:02.790+08:00, sub_task_instance_id=10000083}, listMap={}}, executor=10.120.0.1 2023-11-27 13:40:03.184 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,2fa4e1bf579d] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-11-27 13:40:03.187 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:40:03.184157+08:00, lastProcessTime=2023-11-27T13:39:52.860767+08:00 2023-11-27 13:40:03.204 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000064, status:SUCCESS 2023-11-27 13:40:03.222 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,b7ed28eb4341] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000079, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:40:03.269 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000085, operation:EXECUTE, state: RUNNING 2023-11-27 13:40:03.269 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000066, status:SUCCESS 2023-11-27 13:40:03.273 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000066, status:SUCCESS 2023-11-27 13:40:03.288 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,5316204ced6d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000084, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000070, task_operation=execute, latest_execution_start_time=2023-11-27T13:40:03.064+08:00, sub_task_instance_id=10000084}, listMap={}}, executor=10.120.0.1 2023-11-27 13:40:03.288 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,5316204ced6d] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2023-11-27 13:40:03.350 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000085, name=High Risk Sql Audit Stat Task, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.HighRiskSqlAuditStatTask, seriesId=0, startTime=2023-11-27T13:40:03.250+08:00, endTime=null} 2023-11-27 13:40:03.416 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000086, operation:EXECUTE, state: RUNNING 2023-11-27 13:40:03.528 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000086, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-11-27T13:40:03.413+08:00, endTime=null} 2023-11-27 13:40:03.568 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,024925d1d03c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000085, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=8000071, task_operation=execute, latest_execution_start_time=2023-11-27T13:40:03.249+08:00, sub_task_instance_id=10000085}, listMap={cluster_ids=[[1,1,2]], tenant_ids=[[3,2,5]]}}, executor=10.120.0.1 2023-11-27 13:40:03.568 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,024925d1d03c] c.o.o.p.s.i.t.HighRiskSqlAuditStatTask : start to diagnose[3, 2, 5] 2023-11-27 13:40:03.577 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,024925d1d03c] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_3currentTimeUs:1701063603568000 2023-11-27 13:40:03.577 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,024925d1d03c] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_3diagnosisStartTimeUs:1701063377710000 2023-11-27 13:40:03.580 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,85e02cdbd805] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obcluster, sql count is 0. 2023-11-27 13:40:03.580 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,85e02cdbd805] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, result is empty. 2023-11-27 13:40:03.582 INFO 10 --- [pool-1237-thread-2,554481c3fd1b426b,06210f87b662] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:darenOB, tenant:daren 2023-11-27 13:40:03.589 INFO 10 --- [pool-1237-thread-1,554481c3fd1b426b,ae7d401a3885] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:darenOB, tenant:sys 2023-11-27 13:40:03.598 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:40:03.184157+08:00 2023-11-27 13:40:03.604 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,024925d1d03c] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_3totalLength:0 2023-11-27 13:40:03.624 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,024925d1d03c] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_2currentTimeUs:1701063603609000 2023-11-27 13:40:03.624 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,024925d1d03c] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_2diagnosisStartTimeUs:1701063373260000 2023-11-27 13:40:03.657 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,ba9ea31eae89] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-11-27 13:40:03.657 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,024925d1d03c] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_2totalLength:0 2023-11-27 13:40:03.660 INFO 10 --- [pool-1238-thread-1,554481c3fd1b426b,973bb8dff1de] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2023-11-27 13:40:03.661 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,ba9ea31eae89] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster darenOB, trans stat result is empty. 2023-11-27 13:40:03.675 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,024925d1d03c] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_5currentTimeUs:1701063603659000 2023-11-27 13:40:03.675 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,024925d1d03c] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_5diagnosisStartTimeUs:1701063378311000 2023-11-27 13:40:03.678 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,cc47c8f39d46] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000086, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000072, task_operation=execute, latest_execution_start_time=2023-11-27T13:40:03.412+08:00, sub_task_instance_id=10000086}, listMap={}}, executor=10.120.0.1 2023-11-27 13:40:03.688 INFO 10 --- [pool-subtask-executor17,554481c3fd1b426b,ba9ea31eae89] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000081, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:40:03.698 INFO 10 --- [pool-1238-thread-1,554481c3fd1b426b,973bb8dff1de] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:03.699 INFO 10 --- [pool-1238-thread-1,554481c3fd1b426b,973bb8dff1de] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:03.699 INFO 10 --- [pool-1238-thread-1,554481c3fd1b426b,973bb8dff1de] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:03.701 INFO 10 --- [pool-1238-thread-1,554481c3fd1b426b,973bb8dff1de] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:03.717 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,85e02cdbd805] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:03.719 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,85e02cdbd805] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:03.719 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,85e02cdbd805] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:03.722 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,85e02cdbd805] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:03.731 INFO 10 --- [pool-1237-thread-1,554481c3fd1b426b,ae7d401a3885] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:03.733 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,024925d1d03c] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_5totalLength:0 2023-11-27 13:40:03.735 INFO 10 --- [pool-1237-thread-1,554481c3fd1b426b,ae7d401a3885] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:03.735 INFO 10 --- [pool-1237-thread-1,554481c3fd1b426b,ae7d401a3885] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:03.736 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,85e02cdbd805] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:03.736 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,85e02cdbd805] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:03.737 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,85e02cdbd805] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-11-27 13:40:03.737 INFO 10 --- [pool-1237-thread-1,554481c3fd1b426b,ae7d401a3885] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:03.743 INFO 10 --- [pool-1237-thread-1,554481c3fd1b426b,ae7d401a3885] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:03.743 INFO 10 --- [pool-1237-thread-1,554481c3fd1b426b,ae7d401a3885] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:03.746 INFO 10 --- [pool-subtask-executor9,554481c3fd1b426b,cc47c8f39d46] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000086, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:40:03.751 INFO 10 --- [pool-1237-thread-1,554481c3fd1b426b,ae7d401a3885] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-11-27 13:40:03.758 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,85e02cdbd805] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for darenOB, save 1 to SqlAggregateData 2023-11-27 13:40:03.759 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,024925d1d03c] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000085, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:40:03.764 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,85e02cdbd805] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000082, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:40:03.770 INFO 10 --- [pool-1237-thread-2,554481c3fd1b426b,06210f87b662] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:03.771 INFO 10 --- [pool-1237-thread-2,554481c3fd1b426b,06210f87b662] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:03.771 INFO 10 --- [pool-1237-thread-2,554481c3fd1b426b,06210f87b662] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:03.772 INFO 10 --- [pool-1237-thread-2,554481c3fd1b426b,06210f87b662] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:03.903 INFO 10 --- [pool-1237-thread-1,554481c3fd1b426b,ae7d401a3885] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1701021601748952808, broadcast scn:1701021601748952808, saved frozen scn:1701021601748952808 2023-11-27 13:40:03.903 INFO 10 --- [pool-1237-thread-1,554481c3fd1b426b,ae7d401a3885] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:darenOB, tenant:sys 2023-11-27 13:40:03.908 INFO 10 --- [pool-1237-thread-2,554481c3fd1b426b,06210f87b662] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:03.908 INFO 10 --- [pool-1237-thread-2,554481c3fd1b426b,06210f87b662] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:03.909 INFO 10 --- [pool-1237-thread-2,554481c3fd1b426b,06210f87b662] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-11-27 13:40:03.953 INFO 10 --- [pool-1237-thread-2,554481c3fd1b426b,06210f87b662] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1701021600396785720, broadcast scn:1701021600396785720, saved frozen scn:1701021600396785720 2023-11-27 13:40:03.953 INFO 10 --- [pool-1237-thread-2,554481c3fd1b426b,06210f87b662] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:darenOB, tenant:daren 2023-11-27 13:40:03.959 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,2fa4e1bf579d] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-11-27 13:40:03.968 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,2fa4e1bf579d] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000083, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:40:04.159 INFO 10 --- [pool-ocp-schedules-7,,42b1b9bd7534] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=585625224722984960 2023-11-27 13:40:04.193 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:40:04.197 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:40:04.236 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:40:04.284 INFO 10 --- [pool-1238-thread-1,554481c3fd1b426b,973bb8dff1de] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:04.284 INFO 10 --- [pool-1238-thread-1,554481c3fd1b426b,973bb8dff1de] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:04.285 INFO 10 --- [pool-1238-thread-1,554481c3fd1b426b,973bb8dff1de] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1700977203000000, 1701063603000000] 2023-11-27 13:40:04.535 INFO 10 --- [alarm-detect-5,,3b2fa96c68a2] c.o.o.s.a.core.detect.MetricDetector : alarmType host_unavailable detect time range 1701063534 ~ 1701063594 matched, alarm-level: 1, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={host_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=host_available, match=true, value=0.0, values={1701063540=0.0}, valueCount=1, labels={app=HOST, host=133.197.206.92, host_name=localhost.localdomain, ob_cluster_id=, ob_cluster_name=, svr_ip=133.197.206.92})]}, matched=true, message=null, level=1) 2023-11-27 13:40:04.535 INFO 10 --- [alarm-detect-5,,3b2fa96c68a2] c.o.o.s.a.core.detect.MetricDetector : alarmType host_unavailable detect time range 1701063534 ~ 1701063594 matched, alarm-level: 1, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={host_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=host_available, match=true, value=0.0, values={1701063540=0.0}, valueCount=1, labels={app=HOST, host=133.197.206.97, host_name=localhost.localdomain, ob_cluster_id=1, ob_cluster_name=obcluster, svr_ip=133.197.206.97})]}, matched=true, message=null, level=1) 2023-11-27 13:40:04.535 INFO 10 --- [alarm-detect-5,,3b2fa96c68a2] c.o.o.s.a.core.detect.MetricDetector : alarmType host_unavailable detect time range 1701063534 ~ 1701063594 matched, alarm-level: 1, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={host_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=host_available, match=true, value=0.0, values={1701063540=0.0}, valueCount=1, labels={app=HOST, host=133.197.206.88, host_name=localhost.localdomain, ob_cluster_id=, ob_cluster_name=, svr_ip=133.197.206.88})]}, matched=true, message=null, level=1) 2023-11-27 13:40:04.535 INFO 10 --- [alarm-detect-5,,3b2fa96c68a2] c.o.o.s.a.core.detect.MetricDetector : alarmType host_unavailable detect time range 1701063534 ~ 1701063594 matched, alarm-level: 1, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={host_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=host_available, match=true, value=0.0, values={1701063540=0.0}, valueCount=1, labels={app=HOST, host=133.197.204.3, host_name=localhost.localdomain, ob_cluster_id=10, ob_cluster_name=darenOB, svr_ip=133.197.204.3})]}, matched=true, message=null, level=1) 2023-11-27 13:40:04.536 INFO 10 --- [alarm-detect-5,,3b2fa96c68a2] c.o.o.s.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=host_unavailable, target=alarm_template_id=0:host=133.197.206.92-1, currentState=Firing 2023-11-27 13:40:04.536 INFO 10 --- [alarm-detect-5,,3b2fa96c68a2] c.o.o.s.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=host_unavailable, target=alarm_template_id=0:host=133.197.206.97-1, currentState=Firing 2023-11-27 13:40:04.536 INFO 10 --- [alarm-detect-5,,3b2fa96c68a2] c.o.o.s.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=host_unavailable, target=alarm_template_id=0:host=133.197.206.88-1, currentState=Firing 2023-11-27 13:40:04.536 INFO 10 --- [alarm-detect-5,,3b2fa96c68a2] c.o.o.s.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=host_unavailable, target=alarm_template_id=0:host=133.197.204.3-1, currentState=Firing 2023-11-27 13:40:04.537 INFO 10 --- [alarm-detect-5,,3b2fa96c68a2] c.o.o.s.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=host_unavailable, needSendingCount=4, detectAlarmsCount=4, matchedTargets=[alarm_template_id=0:host=133.197.206.97-1, alarm_template_id=0:host=133.197.206.88-1, alarm_template_id=0:host=133.197.206.92-1, alarm_template_id=0:host=133.197.204.3-1] 2023-11-27 13:40:04.552 INFO 10 --- [alarm-detect-2,,210d6404fe55] c.o.o.s.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans, startId: 348000001, data is empty. 2023-11-27 13:40:04.557 INFO 10 --- [alarm-detect-2,,210d6404fe55] c.o.o.s.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans, startId: 348000002, data is empty. 2023-11-27 13:40:04.558 WARN 10 --- [pool-1238-thread-1,554481c3fd1b426b,973bb8dff1de] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:darenOB, ob tenant id:1001 2023-11-27 13:40:04.574 INFO 10 --- [alarm-detect-12,,e3a90a970354] c.o.o.s.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist, startId: 348000001, data is empty. 2023-11-27 13:40:04.586 INFO 10 --- [alarm-detect-12,,e3a90a970354] c.o.o.s.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist, startId: 348000002, data is empty. 2023-11-27 13:40:04.613 INFO 10 --- [alarm-detect-4,,4cbb393af8f1] c.o.o.s.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist, startId: 348000001, data is empty. 2023-11-27 13:40:04.626 INFO 10 --- [alarm-detect-4,,4cbb393af8f1] c.o.o.s.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist, startId: 348000002, data is empty. 2023-11-27 13:40:04.628 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000076, operation:EXECUTE, state: RUNNING 2023-11-27 13:40:04.635 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000076, name=Sync system event, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask, seriesId=1, startTime=2023-11-27T13:40:04.624+08:00, endTime=null} 2023-11-27 13:40:04.651 INFO 10 --- [alarm-detect-8,,20d50f376ef3] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063534 ~ 1701063594 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063540=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/obproxy, exporter_addr=http://133.197.204.3:62889/metrics/obproxy, exporter_type=OB_PROXY, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:40:04.652 INFO 10 --- [alarm-detect-8,,20d50f376ef3] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063534 ~ 1701063594 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063540=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/ob/basic, exporter_addr=http://133.197.206.97:62889/metrics/ob/basic, exporter_type=OB_CLUSTER, host=133.197.206.97, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.206.97})]}, matched=true, message=null, level=3) 2023-11-27 13:40:04.652 INFO 10 --- [alarm-detect-8,,20d50f376ef3] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063534 ~ 1701063594 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063540=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/ob, exporter_addr=http://133.197.206.97:62889/metrics/node/ob, exporter_type=OB_CLUSTER, host=133.197.206.97, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.206.97})]}, matched=true, message=null, level=3) 2023-11-27 13:40:04.652 INFO 10 --- [alarm-detect-8,,20d50f376ef3] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063534 ~ 1701063594 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063540=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://133.197.206.92:62889/metrics/stat, exporter_type=HOST_MONITOR, host=133.197.206.92, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.206.92})]}, matched=true, message=null, level=3) 2023-11-27 13:40:04.652 INFO 10 --- [alarm-detect-8,,20d50f376ef3] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063534 ~ 1701063594 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063540=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/ob/basic, exporter_addr=http://133.197.204.3:62889/metrics/ob/basic, exporter_type=OB_CLUSTER, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:40:04.652 INFO 10 --- [alarm-detect-8,,20d50f376ef3] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063534 ~ 1701063594 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063540=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://133.197.204.3:62888/metrics/stat, exporter_type=HOST_MONITOR, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:40:04.652 INFO 10 --- [alarm-detect-8,,20d50f376ef3] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063534 ~ 1701063594 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063540=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/ob/extra, exporter_addr=http://133.197.206.97:62889/metrics/ob/extra, exporter_type=OB_CLUSTER, host=133.197.206.97, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.206.97})]}, matched=true, message=null, level=3) 2023-11-27 13:40:04.653 INFO 10 --- [alarm-detect-8,,20d50f376ef3] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063534 ~ 1701063594 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063540=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://133.197.206.97:62888/metrics/stat, exporter_type=HOST_MONITOR, host=133.197.206.97, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.206.97})]}, matched=true, message=null, level=3) 2023-11-27 13:40:04.653 INFO 10 --- [alarm-detect-8,,20d50f376ef3] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063534 ~ 1701063594 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063540=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://133.197.206.92:62888/metrics/stat, exporter_type=HOST_MONITOR, host=133.197.206.92, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.206.92})]}, matched=true, message=null, level=3) 2023-11-27 13:40:04.653 INFO 10 --- [alarm-detect-8,,20d50f376ef3] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063534 ~ 1701063594 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063540=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/host, exporter_addr=http://133.197.204.3:62889/metrics/node/host, exporter_type=HOST_MONITOR, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:40:04.653 INFO 10 --- [alarm-detect-8,,20d50f376ef3] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063534 ~ 1701063594 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063540=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://133.197.204.3:62889/metrics/stat, exporter_type=HOST_MONITOR, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:40:04.653 INFO 10 --- [alarm-detect-8,,20d50f376ef3] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063534 ~ 1701063594 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063540=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://133.197.206.88:62888/metrics/stat, exporter_type=HOST_MONITOR, host=133.197.206.88, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.206.88})]}, matched=true, message=null, level=3) 2023-11-27 13:40:04.653 INFO 10 --- [alarm-detect-8,,20d50f376ef3] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063534 ~ 1701063594 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063540=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/host, exporter_addr=http://133.197.206.92:62889/metrics/node/host, exporter_type=HOST_MONITOR, host=133.197.206.92, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.206.92})]}, matched=true, message=null, level=3) 2023-11-27 13:40:04.653 INFO 10 --- [alarm-detect-8,,20d50f376ef3] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063534 ~ 1701063594 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063540=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://133.197.206.97:62889/metrics/stat, exporter_type=HOST_MONITOR, host=133.197.206.97, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.206.97})]}, matched=true, message=null, level=3) 2023-11-27 13:40:04.654 INFO 10 --- [alarm-detect-8,,20d50f376ef3] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063534 ~ 1701063594 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063540=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/host, exporter_addr=http://133.197.206.88:62889/metrics/node/host, exporter_type=HOST_MONITOR, host=133.197.206.88, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.206.88})]}, matched=true, message=null, level=3) 2023-11-27 13:40:04.654 INFO 10 --- [alarm-detect-8,,20d50f376ef3] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063534 ~ 1701063594 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063540=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/ob, exporter_addr=http://133.197.204.3:62889/metrics/node/ob, exporter_type=OB_CLUSTER, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:40:04.654 INFO 10 --- [alarm-detect-8,,20d50f376ef3] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063534 ~ 1701063594 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063540=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/ob/extra, exporter_addr=http://133.197.204.3:62889/metrics/ob/extra, exporter_type=OB_CLUSTER, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:40:04.654 INFO 10 --- [alarm-detect-8,,20d50f376ef3] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063534 ~ 1701063594 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063540=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/host, exporter_addr=http://133.197.206.97:62889/metrics/node/host, exporter_type=HOST_MONITOR, host=133.197.206.97, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.206.97})]}, matched=true, message=null, level=3) 2023-11-27 13:40:04.654 INFO 10 --- [alarm-detect-8,,20d50f376ef3] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063534 ~ 1701063594 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063540=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/obproxy, exporter_addr=http://133.197.204.3:62889/metrics/node/obproxy, exporter_type=OB_PROXY, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:40:04.654 INFO 10 --- [alarm-detect-8,,20d50f376ef3] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063534 ~ 1701063594 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063540=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://133.197.206.88:62889/metrics/stat, exporter_type=HOST_MONITOR, host=133.197.206.88, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.206.88})]}, matched=true, message=null, level=3) 2023-11-27 13:40:04.654 INFO 10 --- [alarm-detect-8,,20d50f376ef3] c.o.o.s.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=monitor_exporter_unavaliable, target=alarm_template_id=0:host=133.197.204.3:exporter=/metrics/obproxy-3, currentState=Firing 2023-11-27 13:40:04.655 INFO 10 --- [alarm-detect-8,,20d50f376ef3] c.o.o.s.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=monitor_exporter_unavaliable, target=alarm_template_id=0:host=133.197.204.3:exporter=/metrics/ob/basic-3, currentState=Firing 2023-11-27 13:40:04.655 INFO 10 --- [alarm-detect-8,,20d50f376ef3] c.o.o.s.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=monitor_exporter_unavaliable, target=alarm_template_id=0:host=133.197.204.3:exporter=/metrics/stat-3, currentState=Firing 2023-11-27 13:40:04.667 INFO 10 --- [alarm-detect-13,,524b7da705e2] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_cluster_exists_inactive_server detect time range 1701063584 ~ 1701063594 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={inactive_server_count > 0=[ExpressionEvaluateResult(matchValue=0.0, metric=inactive_server_count, match=true, value=1.0, values={1701063590=1.0}, valueCount=1, labels={app=OB, ob_cluster_id=10, ob_cluster_name=darenOB, server_ips=133.197.204.3})]}, matched=true, message=null, level=2) 2023-11-27 13:40:04.667 INFO 10 --- [alarm-detect-5,,3b2fa96c68a2] c.o.o.s.a.service.OcpAlarmServiceImpl : create alarm event Done, affectRow=1, id=2000005 2023-11-27 13:40:04.671 INFO 10 --- [alarm-detect-8,,20d50f376ef3] c.o.o.s.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=monitor_exporter_unavaliable, target=alarm_template_id=0:host=133.197.204.3:exporter=/metrics/node/host-3, currentState=Firing 2023-11-27 13:40:04.672 INFO 10 --- [alarm-detect-8,,20d50f376ef3] c.o.o.s.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=monitor_exporter_unavaliable, target=alarm_template_id=0:host=133.197.204.3:exporter=/metrics/node/ob-3, currentState=Firing 2023-11-27 13:40:04.672 INFO 10 --- [alarm-detect-8,,20d50f376ef3] c.o.o.s.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=monitor_exporter_unavaliable, target=alarm_template_id=0:host=133.197.204.3:exporter=/metrics/ob/extra-3, currentState=Firing 2023-11-27 13:40:04.672 INFO 10 --- [alarm-detect-8,,20d50f376ef3] c.o.o.s.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=monitor_exporter_unavaliable, target=alarm_template_id=0:host=133.197.204.3:exporter=/metrics/node/obproxy-3, currentState=Firing 2023-11-27 13:40:04.672 INFO 10 --- [alarm-detect-5,,3b2fa96c68a2] c.o.o.s.a.service.OcpAlarmServiceImpl : create alarm event history Done, affectRow=1, id=2000005 2023-11-27 13:40:04.672 INFO 10 --- [alarm-detect-8,,20d50f376ef3] c.o.o.s.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=monitor_exporter_unavaliable, needSendingCount=7, detectAlarmsCount=7, matchedTargets=[alarm_template_id=0:host=133.197.204.3:exporter=/metrics/node/host-3, alarm_template_id=0:host=133.197.204.3:exporter=/metrics/stat-3, alarm_template_id=0:host=133.197.206.88:exporter=/metrics/stat-3, alarm_template_id=0:host=133.197.206.97:exporter=/metrics/node/ob-3, alarm_template_id=0:host=133.197.206.92:exporter=/metrics/stat-3, alarm_template_id=0:host=133.197.204.3:exporter=/metrics/ob/basic-3, alarm_template_id=0:host=133.197.206.97:exporter=/metrics/stat-3, alarm_template_id=0:host=133.197.204.3:exporter=/metrics/ob/extra-3, alarm_template_id=0:host=133.197.206.97:exporter=/metrics/node/host-3, alarm_template_id=0:host=133.197.204.3:exporter=/metrics/node/ob-3, alarm_template_id=0:host=133.197.206.88:exporter=/metrics/node/host-3, alarm_template_id=0:host=133.197.204.3:exporter=/metrics/node/obproxy-3, alarm_template_id=0:host=133.197.206.97:exporter=/metrics/ob/extra-3, alarm_template_id=0:host=133.197.204.3:exporter=/metrics/obproxy-3, alarm_template_id=0:host=133.197.206.97:exporter=/metrics/ob/basic-3, alarm_template_id=0:host=133.197.206.92:exporter=/metrics/node/host-3] 2023-11-27 13:40:04.701 INFO 10 --- [pool-1238-thread-1,554481c3fd1b426b,973bb8dff1de] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2023-11-27 13:40:04.703 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,5316204ced6d] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2023-11-27 13:40:04.704 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000077, operation:EXECUTE, state: RUNNING 2023-11-27 13:40:04.716 INFO 10 --- [alarm-detect-5,,3b2fa96c68a2] c.o.o.s.a.service.OcpAlarmServiceImpl : create alarm event Done, affectRow=1, id=2000006 2023-11-27 13:40:04.716 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000077, name=Sync system event, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask, seriesId=2, startTime=2023-11-27T13:40:04.699+08:00, endTime=null} 2023-11-27 13:40:04.727 INFO 10 --- [alarm-detect-5,,3b2fa96c68a2] c.o.o.s.a.service.OcpAlarmServiceImpl : create alarm event history Done, affectRow=1, id=2000006 2023-11-27 13:40:04.737 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,5316204ced6d] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000084, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:40:04.747 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,09620676b3ff] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000076, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=8000063, task_operation=execute, latest_execution_start_time=2023-11-27T13:40:04.624+08:00, sub_task_instance_id=10000076}, listMap={cluster_ids=[1, 2]}}, executor=10.120.0.1 2023-11-27 13:40:04.782 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000065, status:SUCCESS 2023-11-27 13:40:04.783 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000065, status:SUCCESS 2023-11-27 13:40:04.783 INFO 10 --- [pool-ocp-schedules-19,,e4b9a81011e6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:40:04.788 INFO 10 --- [alarm-detect-5,,3b2fa96c68a2] c.o.o.s.a.service.OcpAlarmServiceImpl : create alarm event Done, affectRow=1, id=2000007 2023-11-27 13:40:04.792 INFO 10 --- [alarm-detect-5,,3b2fa96c68a2] c.o.o.s.a.service.OcpAlarmServiceImpl : create alarm event history Done, affectRow=1, id=2000007 2023-11-27 13:40:04.803 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000067, status:SUCCESS 2023-11-27 13:40:04.805 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000067, status:SUCCESS 2023-11-27 13:40:04.811 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c7015705848e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000077, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=8000063, task_operation=execute, latest_execution_start_time=2023-11-27T13:40:04.698+08:00, sub_task_instance_id=10000077}, listMap={cluster_ids=[1, 2]}}, executor=10.120.0.1 2023-11-27 13:40:04.834 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:36:06.0, nextRun=2023-11-27 13:40:04.0, currentDate=Mon Nov 27 13:40:04 CST 2023 2023-11-27 13:40:04.835 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=1, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2023-11-27 13:36:06.0, nextRunTime=2023-11-27 13:40:04.0} 2023-11-27 13:40:04.880 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000068, status:SUCCESS 2023-11-27 13:40:04.882 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000068, status:SUCCESS 2023-11-27 13:40:04.927 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000069, status:SUCCESS 2023-11-27 13:40:04.929 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000069, status:SUCCESS 2023-11-27 13:40:05.025 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2023-11-27 13:40:05.025 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:40:05.025 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Partition rollover daemon, taskDefinitionId=1, state=RUNNING, operation=EXECUTE} 2023-11-27 13:40:05.070 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000070, status:SUCCESS 2023-11-27 13:40:05.071 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000070, status:SUCCESS 2023-11-27 13:40:05.102 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063605, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:05.103 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063605, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:05.103 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063605, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:05.104 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063605, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:05.104 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063605, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:05.160 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:39:06.0, nextRun=2023-11-27 13:40:05.0, currentDate=Mon Nov 27 13:40:05 CST 2023 2023-11-27 13:40:05.161 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-11-27 13:39:06.0, nextRunTime=2023-11-27 13:40:05.0} 2023-11-27 13:40:05.165 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-11-27 13:40:05.165 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:40:05.165 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-11-27 13:40:05.174 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000071, status:SUCCESS 2023-11-27 13:40:05.176 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000071, status:SUCCESS 2023-11-27 13:40:05.205 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000072, status:SUCCESS 2023-11-27 13:40:05.208 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000072, status:SUCCESS 2023-11-27 13:40:05.290 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:40:05.413 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:40:05.414 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:40:05.775 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000088, operation:EXECUTE, state: RUNNING 2023-11-27 13:40:05.778 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:40:05.780 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000088, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-11-27T13:40:05.766+08:00, endTime=null} 2023-11-27 13:40:05.798 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000087, operation:EXECUTE, state: RUNNING 2023-11-27 13:40:05.804 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000087, name=Partition rollover daemon, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=2023-11-27T13:40:05.796+08:00, endTime=null} 2023-11-27 13:40:05.812 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,26059198af43] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000088, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000074, task_operation=execute, latest_execution_start_time=2023-11-27T13:40:05.765+08:00, sub_task_instance_id=10000088}, listMap={}}, executor=10.120.0.1 2023-11-27 13:40:05.851 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000087, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000073, task_operation=execute, latest_execution_start_time=2023-11-27T13:40:05.795+08:00, sub_task_instance_id=10000087}, listMap={}}, executor=10.120.0.1 2023-11-27 13:40:05.864 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2023-11-27 13:40:05.864 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2023-11-27 13:40:05.866 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:05.866 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:05.866 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:05.867 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:05.868 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:05.868 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2023-11-27 13:40:06.365 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:40:06.541 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.541 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.542 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:06.551 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.551 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.552 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2023-11-27 13:40:06.570 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.570 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.570 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:06.571 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.572 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.572 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2023-11-27 13:40:06.588 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.588 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.588 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:06.589 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.589 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.590 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2023-11-27 13:40:06.608 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.608 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.609 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:06.609 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.609 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.610 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2023-11-27 13:40:06.617 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.617 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.618 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:06.619 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.619 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.619 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2023-11-27 13:40:06.628 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.628 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.628 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:06.629 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.629 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.630 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2023-11-27 13:40:06.636 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.637 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.637 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:06.638 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.638 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.638 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2023-11-27 13:40:06.647 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.648 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.648 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:06.649 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.650 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.651 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2023-11-27 13:40:06.656 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.657 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.657 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:06.658 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.658 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.658 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2023-11-27 13:40:06.671 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.671 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.672 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:06.673 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.674 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.674 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2023-11-27 13:40:06.699 INFO 10 --- [pool-Check obproxy10,554481c3fd1b426b,a024c6d0003a] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-11-27 13:40:06.700 INFO 10 --- [pool-Check obproxy6,554481c3fd1b426b,f4b1cc10d00e] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-11-27 13:40:06.702 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.702 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.703 WARN 10 --- [pool-Check obproxy6,554481c3fd1b426b,f4b1cc10d00e] c.oceanbase.ocp.odpsdk.OdpJdbcOperator : Failed to check odp connection Connection {driverClassName ='com.oceanbase.jdbc.Driver', connectionProperties ='[user=root@proxysys, url=jdbc:oceanbase://133.197.204.3:2883/?connectProxy=true]'}, error is Failed to obtain JDBC Connection; nested exception is java.sql.SQLNonTransientConnectionException: Could not connect to HostAddress{host='133.197.204.3', port=2883}. Connection refused (Connection refused) 2023-11-27 13:40:06.703 WARN 10 --- [pool-Check obproxy6,554481c3fd1b426b,f4b1cc10d00e] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to connect obproxy server, maybe caused by odp process down, wrong password, or ocp address not in private ip list and odp parameter skip_proxy_sys_private_check is not true 2023-11-27 13:40:06.703 ERROR 10 --- [pool-Check obproxy6,554481c3fd1b426b,f4b1cc10d00e] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.UnexpectedException occurred with code error.obproxy.connection.error, and args [133.197.204.3:2883] 2023-11-27 13:40:06.704 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:06.707 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.707 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.709 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2023-11-27 13:40:06.711 INFO 10 --- [pool-Check obproxy7,554481c3fd1b426b,6174f6bf069d] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-11-27 13:40:06.730 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.730 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.733 INFO 10 --- [pool-Check obproxy10,554481c3fd1b426b,a024c6d0003a] c.o.o.s.o.obproxy.ObproxyService : check obproxy 11 success, update last active time to now 2023-11-27 13:40:06.733 WARN 10 --- [pool-Check obproxy6,554481c3fd1b426b,f4b1cc10d00e] c.oceanbase.ocp.odpsdk.OdpJdbcOperator : Failed to check odp connection Connection {driverClassName ='com.oceanbase.jdbc.Driver', connectionProperties ='[user=root@proxysys, url=jdbc:oceanbase://133.197.204.3:2883/?connectProxy=true]'}, error is Failed to obtain JDBC Connection; nested exception is java.sql.SQLNonTransientConnectionException: Could not connect to HostAddress{host='133.197.204.3', port=2883}. Connection refused (Connection refused) 2023-11-27 13:40:06.733 WARN 10 --- [pool-Check obproxy6,554481c3fd1b426b,f4b1cc10d00e] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to connect obproxy server, maybe caused by odp process down, wrong password, or ocp address not in private ip list and odp parameter skip_proxy_sys_private_check is not true 2023-11-27 13:40:06.734 ERROR 10 --- [pool-Check obproxy6,554481c3fd1b426b,f4b1cc10d00e] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.UnexpectedException occurred with code error.obproxy.connection.error, and args [133.197.204.3:2883] 2023-11-27 13:40:06.736 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:06.740 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.740 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.745 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2023-11-27 13:40:06.762 INFO 10 --- [pool-Check obproxy7,554481c3fd1b426b,6174f6bf069d] c.o.o.s.o.obproxy.ObproxyService : check obproxy 12 success, update last active time to now 2023-11-27 13:40:06.797 INFO 10 --- [pool-Check obproxy8,554481c3fd1b426b,2ac982379f19] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-11-27 13:40:06.798 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.799 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.814 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:06.817 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.817 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.817 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2023-11-27 13:40:06.828 INFO 10 --- [pool-Check obproxy8,554481c3fd1b426b,2ac982379f19] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-11-27 13:40:06.842 INFO 10 --- [pool-Check obproxy9,554481c3fd1b426b,9029fb77e168] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-11-27 13:40:06.843 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.844 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.848 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:06.852 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.853 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.853 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2023-11-27 13:40:06.853 INFO 10 --- [pool-Check obproxy6,554481c3fd1b426b,f4b1cc10d00e] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 failed, will not update last active time com.oceanbase.ocp.core.exception.UnexpectedException: [OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=OBPROXY_CONNECTION_ERROR, args=133.197.204.3:2883 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_312] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_312] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_312] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_312] at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.util.ExceptionUtils.throwException(ExceptionUtils.java:90) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.util.ExceptionUtils.unExpected(ExceptionUtils.java:77) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.service.operation.obproxy.SystemOdpOperatorFactory.createOdpOperator(SystemOdpOperatorFactory.java:105) ~[ocp-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.service.operation.obproxy.SystemOdpOperatorFactory.tryToCreateOdpOperator(SystemOdpOperatorFactory.java:86) ~[ocp-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.service.operation.obproxy.ObproxyService.lambda$checkObproxyServers$20(ObproxyService.java:517) ~[ocp-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] 2023-11-27 13:40:06.862 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.863 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.863 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:06.864 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.864 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.865 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2023-11-27 13:40:06.875 INFO 10 --- [pool-Check obproxy9,554481c3fd1b426b,9029fb77e168] c.o.o.s.o.obproxy.ObproxyService : check obproxy 10 success, update last active time to now 2023-11-27 13:40:06.888 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,26059198af43] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000088, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:40:06.890 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.891 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.892 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:06.895 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.895 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.895 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2023-11-27 13:40:06.909 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.909 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.910 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:06.910 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.910 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.911 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2023-11-27 13:40:06.918 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.918 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.919 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:06.919 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.919 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.920 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2023-11-27 13:40:06.926 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.926 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.926 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:06.927 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.927 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.927 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2023-11-27 13:40:06.934 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.934 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.934 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:06.935 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.935 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.936 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2023-11-27 13:40:06.956 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.956 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.957 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:06.957 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.957 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.958 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2023-11-27 13:40:06.968 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.969 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.969 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:06.970 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.970 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.970 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2023-11-27 13:40:06.978 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.978 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.979 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:06.980 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.980 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.980 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2023-11-27 13:40:06.987 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.987 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.987 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:06.989 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.989 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.989 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2023-11-27 13:40:06.997 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.997 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.997 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:06.998 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:06.998 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:06.999 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2023-11-27 13:40:07.006 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.006 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.007 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:07.008 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.008 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.008 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2023-11-27 13:40:07.015 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.015 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.016 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:07.017 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.017 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.017 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2023-11-27 13:40:07.027 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.027 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.028 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:07.029 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.029 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.029 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2023-11-27 13:40:07.036 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.036 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.037 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:07.038 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.038 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.039 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2023-11-27 13:40:07.044 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.044 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.045 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:07.046 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.046 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.047 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2023-11-27 13:40:07.056 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.056 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.056 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:07.057 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.057 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.058 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2023-11-27 13:40:07.064 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.064 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.065 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:07.066 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.066 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.066 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2023-11-27 13:40:07.077 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.077 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.078 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:07.079 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.079 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.079 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat 2023-11-27 13:40:07.088 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.088 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.089 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:07.090 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.090 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.090 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2023-11-27 13:40:07.104 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.104 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.105 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:07.106 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.106 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.107 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2023-11-27 13:40:07.112 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.112 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.112 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:07.113 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.113 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.114 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2023-11-27 13:40:07.119 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.119 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.120 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:07.121 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.121 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.121 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2023-11-27 13:40:07.127 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.127 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.127 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:07.128 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.128 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.129 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2023-11-27 13:40:07.133 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.133 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.133 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:07.134 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.134 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.135 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2023-11-27 13:40:07.155 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.155 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.156 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:07.157 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.157 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.158 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2023-11-27 13:40:07.163 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.163 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.164 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:07.165 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.165 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.165 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2023-11-27 13:40:07.172 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.172 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.173 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:07.173 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.173 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.174 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2023-11-27 13:40:07.180 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.180 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.181 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:07.182 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.182 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.182 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2023-11-27 13:40:07.188 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.188 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.189 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:07.190 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.190 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.190 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2023-11-27 13:40:07.196 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.196 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.196 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:07.197 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:40:07.197 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.197 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.198 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2023-11-27 13:40:07.198 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:40:07.203 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2023-11-27 13:40:07.204 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2023-11-27 13:40:07.205 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.205 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.205 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:07.206 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.206 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.207 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2023-11-27 13:40:07.231 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.232 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.232 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-27 13:40:07.233 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2883 2023-11-27 13:40:07.233 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.234 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2023-11-27 13:40:07.265 INFO 10 --- [http-nio-0.0.0.0-8080-exec-1,9e3d24bfe5be436c,e9af3429f80a] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.13, traceId=9e3d24bfe5be436c, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:40:07.354 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2023-11-27 13:40:07.355 INFO 10 --- [http-nio-0.0.0.0-8080-exec-1,9e3d24bfe5be436c,e9af3429f80a] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.13, traceId=9e3d24bfe5be436c, duration=90 ms] 2023-11-27 13:40:07.372 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,c7ff58cfd633] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000087, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:40:07.484 INFO 10 --- [pool-ocp-schedules-1,,04605bb3b0bc] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=11 2023-11-27 13:40:07.586 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,09620676b3ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:07.587 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,09620676b3ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:07.587 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,09620676b3ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.587 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,09620676b3ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:07.595 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,09620676b3ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:07.596 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,09620676b3ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.597 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,09620676b3ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ * FROM oceanbase.__all_rootservice_event_history WHERE 1=1 AND not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND gmt_create > :startTime AND gmt_create <= :endTime ORDER BY gmt_create ASC LIMIT :limit, args: [2023-11-27 13:33:08.122149, 2023-11-27 13:40:07.548, 1000] 2023-11-27 13:40:07.598 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,09620676b3ff] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1, startTime 2023-11-27 13:33:08.122149, endTime 2023-11-27 13:40:07.548 2023-11-27 13:40:07.615 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,09620676b3ff] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000076, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:40:07.624 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c7015705848e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:07.625 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c7015705848e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:07.625 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c7015705848e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.625 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c7015705848e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:07.630 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c7015705848e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:07.630 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c7015705848e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:07.631 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c7015705848e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND timestamp > :startTime AND timestamp <= :endTime ORDER BY timestamp ASC LIMIT :limit, args: [2023-11-27 07:19:59.114835, 2023-11-27 13:40:07.571, 1000] 2023-11-27 13:40:07.632 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c7015705848e] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 2, startTime 2023-11-27 07:19:59.114835, endTime 2023-11-27 13:40:07.571 2023-11-27 13:40:07.639 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,c7015705848e] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000077, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:40:07.729 INFO 10 --- [pool-ocp-schedules-1,,04605bb3b0bc] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=11 2023-11-27 13:40:07.833 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000063, status:SUCCESS 2023-11-27 13:40:07.834 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000063, status:SUCCESS 2023-11-27 13:40:07.872 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000073, status:SUCCESS 2023-11-27 13:40:07.874 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000073, status:SUCCESS 2023-11-27 13:40:07.898 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000074, status:SUCCESS 2023-11-27 13:40:07.900 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000074, status:SUCCESS 2023-11-27 13:40:09.200 INFO 10 --- [http-nio-0.0.0.0-8080-exec-3,33066e9a809d4cc8,44ad6ac87c93] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=133.197.206.97, traceId=33066e9a809d4cc8, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-11-27 13:40:09.212 INFO 10 --- [http-nio-0.0.0.0-8080-exec-3,33066e9a809d4cc8,44ad6ac87c93] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.206.97, traceId=33066e9a809d4cc8, duration=12 ms] 2023-11-27 13:40:09.702 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.120.0.1:8080 2023-11-27 13:40:10.103 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063610, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:10.103 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063610, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:10.103 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063610, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:10.108 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063610, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:10.108 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063610, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:10.198 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:40:10.200 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:40:11.799 INFO 10 --- [pool-overview-container-2,,e8ec7d5494df] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=420, alarmHistory size=15892 2023-11-27 13:40:11.868 INFO 10 --- [pool-scanner--12543-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=76 2023-11-27 13:40:12.792 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:40:12.789260+08:00, lastProcessTime=2023-11-27T13:40:03.184157+08:00 2023-11-27 13:40:12.893 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.ChannelAggregator : aggregate, pre groupAggregator is null, reset it, channelId=100000, groupByKey=0:host_unavailable:1, alarmId=2000005, alarmType=host_unavailable, alarmTarget=alarm_template_id=0:host=133.197.206.97 2023-11-27 13:40:12.894 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.ChannelAggregator : aggregate, pre groupAggregator is null, reset it, channelId=100000, groupByKey=0:monitor_exporter_unavaliable:3, alarmId=44801, alarmType=monitor_exporter_unavaliable, alarmTarget=alarm_template_id=0:host=133.197.204.3:exporter=/metrics/node/host 2023-11-27 13:40:12.897 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.ChannelAggregator : aggregate, pre groupAggregator is null, reset it, channelId=100001, groupByKey=0:host_unavailable:1, alarmId=2000005, alarmType=host_unavailable, alarmTarget=alarm_template_id=0:host=133.197.206.97 2023-11-27 13:40:12.897 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.ChannelAggregator : aggregate, pre groupAggregator is null, reset it, channelId=100001, groupByKey=0:monitor_exporter_unavaliable:3, alarmId=44801, alarmType=monitor_exporter_unavaliable, alarmTarget=alarm_template_id=0:host=133.197.204.3:exporter=/metrics/node/host 2023-11-27 13:40:12.898 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:40:12.789260+08:00 2023-11-27 13:40:13.200 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:40:13.201 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:40:13.399 INFO 10 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1701063540, nodeCount=585, affectRows=585 2023-11-27 13:40:13.402 INFO 10 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1701063540, nodeCount=585, validCount=585 2023-11-27 13:40:13.437 INFO 10 --- [pool-metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1701063540, nodeCount=531, affectRows=531 2023-11-27 13:40:13.439 INFO 10 --- [pool-metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1701063540, nodeCount=531, validCount=531 2023-11-27 13:40:13.452 INFO 10 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1701063540, nodeCount=591, affectRows=591 2023-11-27 13:40:13.456 INFO 10 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1701063540, nodeCount=591, validCount=591 2023-11-27 13:40:14.235 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:40:14.310 INFO 10 --- [http-nio-0.0.0.0-8080-exec-5,5687c2c380f7442b,35769d8e77d9] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.13, traceId=5687c2c380f7442b, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:40:14.370 INFO 10 --- [http-nio-0.0.0.0-8080-exec-5,5687c2c380f7442b,35769d8e77d9] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.13, traceId=5687c2c380f7442b, duration=61 ms] 2023-11-27 13:40:14.509 INFO 10 --- [alarm-detect-8,,082fc863cad3] c.o.o.s.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist, startId: 348000001, data is empty. 2023-11-27 13:40:14.520 INFO 10 --- [alarm-detect-8,,082fc863cad3] c.o.o.s.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist, startId: 348000002, data is empty. 2023-11-27 13:40:14.539 INFO 10 --- [alarm-detect-7,,4d785739a8ad] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_cluster_exists_inactive_server detect time range 1701063594 ~ 1701063604 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={inactive_server_count > 0=[ExpressionEvaluateResult(matchValue=0.0, metric=inactive_server_count, match=true, value=1.0, values={1701063600=1.0}, valueCount=1, labels={app=OB, ob_cluster_id=10, ob_cluster_name=darenOB, server_ips=133.197.204.3})]}, matched=true, message=null, level=2) 2023-11-27 13:40:14.848 INFO 10 --- [pool-ocp-schedules-19,,c805bd538336] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:40:15.104 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063615, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:15.107 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063615, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:15.107 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063615, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:15.107 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063615, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:15.107 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063615, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:15.414 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:40:15.434 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:40:15.466 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:40:15.627 INFO 10 --- [pool-ocp-schedules-4,,a46ea5694b42] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-11-27 13:40:15.628 INFO 10 --- [pool-Check host status6,,cf8b803a7a04] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=133.197.206.97, originalStatus=OFFLINE 2023-11-27 13:40:15.629 INFO 10 --- [pool-Check host status8,,523372fa1ca6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=133.197.204.3, originalStatus=OFFLINE 2023-11-27 13:40:15.629 INFO 10 --- [pool-Check host status13,,517793fb8d2e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=133.197.206.92, originalStatus=OFFLINE 2023-11-27 13:40:15.642 INFO 10 --- [pool-Check host status4,,8fdf4cc5f47b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=133.197.204.10, originalStatus=ONLINE 2023-11-27 13:40:15.642 INFO 10 --- [pool-Check host status7,,6bba02d8f7f0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=6, ip=133.197.204.12, originalStatus=ONLINE 2023-11-27 13:40:15.642 INFO 10 --- [pool-Check host status5,,5341e2a9fb7e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=133.197.204.11, originalStatus=ONLINE 2023-11-27 13:40:15.642 INFO 10 --- [pool-Check host status10,,e7728d7b8d66] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=133.197.204.13, originalStatus=ONLINE 2023-11-27 13:40:15.643 INFO 10 --- [pool-Check host status17,,14df9ca97038] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=9, ip=133.197.206.88, originalStatus=OFFLINE 2023-11-27 13:40:15.657 INFO 10 --- [pool-Check host status4,,8fdf4cc5f47b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-11-27 13:40:15.659 INFO 10 --- [pool-Check host status7,,6bba02d8f7f0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2023-11-27 13:40:15.662 INFO 10 --- [pool-Check host status5,,5341e2a9fb7e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-11-27 13:40:15.663 INFO 10 --- [pool-Check host status13,,517793fb8d2e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:40:15.671 INFO 10 --- [pool-Check host status10,,e7728d7b8d66] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2023-11-27 13:40:15.680 INFO 10 --- [pool-Check host status6,,cf8b803a7a04] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:40:15.681 INFO 10 --- [pool-Check host status8,,523372fa1ca6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:40:15.694 INFO 10 --- [pool-Check host status17,,14df9ca97038] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:40:15.796 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:40:15.842 INFO 10 --- [pool-Check host status13,,517793fb8d2e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.92:62888/api/v1/time, params:null 2023-11-27 13:40:15.844 WARN 10 --- [pool-Check host status13,,517793fb8d2e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check failed. host=3, ip=133.197.206.92, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.92:62888 [/133.197.206.92] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.getHostTime(AgentExecutor.java:127) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.doCheckHostStatus(HostCheckManagerImpl.java:206) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostStatus(HostCheckManagerImpl.java:168) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.92:62888 [/133.197.206.92] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 24 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 33 common frames omitted 2023-11-27 13:40:15.857 INFO 10 --- [pool-Check host status13,,517793fb8d2e] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [3:133.197.206.92], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:40:15.857 INFO 10 --- [pool-Check host status13,,517793fb8d2e] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:40:15.861 INFO 10 --- [pool-Check host status17,,14df9ca97038] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.88:62888/api/v1/time, params:null 2023-11-27 13:40:15.863 INFO 10 --- [pool-Check host status4,,8fdf4cc5f47b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.10:62888/api/v1/time, params:null 2023-11-27 13:40:15.863 INFO 10 --- [pool-Check host status6,,cf8b803a7a04] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.97:62888/api/v1/time, params:null 2023-11-27 13:40:15.864 WARN 10 --- [pool-Check host status17,,14df9ca97038] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check failed. host=9, ip=133.197.206.88, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.88:62888 [/133.197.206.88] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.getHostTime(AgentExecutor.java:127) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.doCheckHostStatus(HostCheckManagerImpl.java:206) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostStatus(HostCheckManagerImpl.java:168) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.88:62888 [/133.197.206.88] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 24 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 33 common frames omitted 2023-11-27 13:40:15.865 WARN 10 --- [pool-Check host status6,,cf8b803a7a04] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check failed. host=1, ip=133.197.206.97, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.97:62888 [/133.197.206.97] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.getHostTime(AgentExecutor.java:127) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.doCheckHostStatus(HostCheckManagerImpl.java:206) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostStatus(HostCheckManagerImpl.java:168) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.97:62888 [/133.197.206.97] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 24 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 33 common frames omitted 2023-11-27 13:40:15.866 INFO 10 --- [pool-Check host status5,,5341e2a9fb7e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.11:62888/api/v1/time, params:null 2023-11-27 13:40:15.869 INFO 10 --- [pool-Check host status4,,8fdf4cc5f47b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=133.197.204.10, timestamp=2023-11-27T00:40:15.860680090-05:00 2023-11-27 13:40:15.869 INFO 10 --- [pool-Check host status5,,5341e2a9fb7e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=133.197.204.11, timestamp=2023-11-27T00:40:15.869574045-05:00 2023-11-27 13:40:15.870 INFO 10 --- [pool-Check host status5,,5341e2a9fb7e] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-11-27T13:40:15.869+08:00 2023-11-27 13:40:15.871 INFO 10 --- [pool-Check host status4,,8fdf4cc5f47b] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-11-27T13:40:15.869+08:00 2023-11-27 13:40:15.872 INFO 10 --- [pool-Check host status13,,517793fb8d2e] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=3 2023-11-27 13:40:15.872 INFO 10 --- [pool-Check host status13,,517793fb8d2e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2023-11-27 13:40:15.873 INFO 10 --- [pool-Check host status17,,14df9ca97038] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [9:133.197.206.88], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:40:15.873 INFO 10 --- [pool-Check host status17,,14df9ca97038] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:40:15.875 INFO 10 --- [pool-Check host status5,,5341e2a9fb7e] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 12 last available time 2023-11-27T13:40:15.869+08:00 and status AVAILABLE 2023-11-27 13:40:15.876 INFO 10 --- [pool-Check host status8,,523372fa1ca6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.3:62888/api/v1/time, params:null 2023-11-27 13:40:15.876 INFO 10 --- [pool-Check host status13,,517793fb8d2e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:40:15.877 INFO 10 --- [pool-Check host status4,,8fdf4cc5f47b] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 11 last available time 2023-11-27T13:40:15.869+08:00 and status AVAILABLE 2023-11-27 13:40:15.878 INFO 10 --- [pool-Check host status6,,cf8b803a7a04] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [1:133.197.206.97], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:40:15.878 INFO 10 --- [pool-Check host status6,,cf8b803a7a04] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:40:15.879 WARN 10 --- [pool-Check host status8,,523372fa1ca6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check failed. host=2, ip=133.197.204.3, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.204.3:62888 [/133.197.204.3] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.getHostTime(AgentExecutor.java:127) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.doCheckHostStatus(HostCheckManagerImpl.java:206) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostStatus(HostCheckManagerImpl.java:168) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.204.3:62888 [/133.197.204.3] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 24 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 33 common frames omitted 2023-11-27 13:40:15.879 INFO 10 --- [pool-Check host status7,,6bba02d8f7f0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.12:62888/api/v1/time, params:null 2023-11-27 13:40:15.881 INFO 10 --- [pool-Check host status17,,14df9ca97038] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=9 2023-11-27 13:40:15.881 INFO 10 --- [pool-Check host status17,,14df9ca97038] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=9 2023-11-27 13:40:15.885 INFO 10 --- [pool-Check host status8,,523372fa1ca6] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [2:133.197.204.3], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:40:15.885 INFO 10 --- [pool-Check host status8,,523372fa1ca6] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:40:15.885 INFO 10 --- [pool-Check host status7,,6bba02d8f7f0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=6, ip=133.197.204.12, timestamp=2023-11-27T00:40:15.880238152-05:00 2023-11-27 13:40:15.886 INFO 10 --- [pool-Check host status17,,14df9ca97038] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:40:15.888 INFO 10 --- [pool-Check host status6,,cf8b803a7a04] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=1 2023-11-27 13:40:15.888 INFO 10 --- [pool-Check host status6,,cf8b803a7a04] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2023-11-27 13:40:15.888 INFO 10 --- [pool-Check host status7,,6bba02d8f7f0] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=6, lastAvailableTime=2023-11-27T13:40:15.885+08:00 2023-11-27 13:40:15.891 INFO 10 --- [pool-Check host status8,,523372fa1ca6] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=2 2023-11-27 13:40:15.891 INFO 10 --- [pool-Check host status8,,523372fa1ca6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2023-11-27 13:40:15.892 INFO 10 --- [pool-Check host status6,,cf8b803a7a04] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:40:15.893 INFO 10 --- [pool-Check host status8,,523372fa1ca6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:40:15.899 INFO 10 --- [pool-Check host status7,,6bba02d8f7f0] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 13 last available time 2023-11-27T13:40:15.885+08:00 and status AVAILABLE 2023-11-27 13:40:15.903 INFO 10 --- [pool-Check host status4,,8fdf4cc5f47b] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-11-27T13:40:15.902+08:00 2023-11-27 13:40:15.907 INFO 10 --- [pool-Check host status10,,e7728d7b8d66] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.13:62888/api/v1/time, params:null 2023-11-27 13:40:15.909 INFO 10 --- [pool-Check host status4,,8fdf4cc5f47b] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 11 last available time 2023-11-27T13:40:15.902+08:00 and status AVAILABLE 2023-11-27 13:40:15.911 INFO 10 --- [pool-Check host status10,,e7728d7b8d66] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=7, ip=133.197.204.13, timestamp=2023-11-27T00:40:15.892955224-05:00 2023-11-27 13:40:15.915 INFO 10 --- [pool-Check host status10,,e7728d7b8d66] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2023-11-27T13:40:15.911+08:00 2023-11-27 13:40:15.920 INFO 10 --- [pool-Check host status5,,5341e2a9fb7e] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-11-27T13:40:15.919+08:00 2023-11-27 13:40:15.921 INFO 10 --- [pool-Check host status10,,e7728d7b8d66] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 14 last available time 2023-11-27T13:40:15.911+08:00 and status AVAILABLE 2023-11-27 13:40:15.926 INFO 10 --- [pool-Check host status4,,8fdf4cc5f47b] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-11-27 13:40:15.932 INFO 10 --- [pool-Check host status5,,5341e2a9fb7e] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 12 last available time 2023-11-27T13:40:15.919+08:00 and status AVAILABLE 2023-11-27 13:40:15.937 INFO 10 --- [pool-Check host status7,,6bba02d8f7f0] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=6, lastAvailableTime=2023-11-27T13:40:15.933+08:00 2023-11-27 13:40:15.942 INFO 10 --- [pool-Check host status7,,6bba02d8f7f0] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 13 last available time 2023-11-27T13:40:15.933+08:00 and status AVAILABLE 2023-11-27 13:40:15.949 INFO 10 --- [pool-Check host status5,,5341e2a9fb7e] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-11-27 13:40:15.952 INFO 10 --- [pool-Check host status7,,6bba02d8f7f0] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=6 2023-11-27 13:40:15.955 INFO 10 --- [pool-Check host status4,,8fdf4cc5f47b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-11-27 13:40:15.956 INFO 10 --- [pool-Check host status10,,e7728d7b8d66] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2023-11-27T13:40:15.955+08:00 2023-11-27 13:40:15.961 INFO 10 --- [pool-Check host status10,,e7728d7b8d66] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 14 last available time 2023-11-27T13:40:15.955+08:00 and status AVAILABLE 2023-11-27 13:40:15.974 INFO 10 --- [pool-Check host status7,,6bba02d8f7f0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2023-11-27 13:40:15.999 INFO 10 --- [pool-Check host status5,,5341e2a9fb7e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-11-27 13:40:16.007 INFO 10 --- [pool-Check host status7,,6bba02d8f7f0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-11-27 13:40:16.012 INFO 10 --- [pool-Check host status10,,e7728d7b8d66] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=7 2023-11-27 13:40:16.028 INFO 10 --- [pool-Check host status4,,8fdf4cc5f47b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-11-27 13:40:16.044 INFO 10 --- [pool-Check host status10,,e7728d7b8d66] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2023-11-27 13:40:16.056 INFO 10 --- [pool-Check host status7,,6bba02d8f7f0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=37 2023-11-27 13:40:16.078 INFO 10 --- [pool-Check host status4,,8fdf4cc5f47b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=31 2023-11-27 13:40:16.088 INFO 10 --- [pool-Check host status5,,5341e2a9fb7e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-11-27 13:40:16.093 INFO 10 --- [pool-Check host status7,,6bba02d8f7f0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=37 2023-11-27 13:40:16.105 INFO 10 --- [pool-Check host status4,,8fdf4cc5f47b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=31 2023-11-27 13:40:16.109 INFO 10 --- [pool-ocp-schedules-13,,367e8f55603a] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:40:16.109 INFO 10 --- [pool-ocp-schedules-13,,367e8f55603a] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:40:16.137 INFO 10 --- [pool-Check host status5,,5341e2a9fb7e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=34 2023-11-27 13:40:16.144 INFO 10 --- [pool-Check host status10,,e7728d7b8d66] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-11-27 13:40:16.170 INFO 10 --- [pool-Check host status7,,6bba02d8f7f0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-11-27 13:40:16.184 INFO 10 --- [pool-Check host status5,,5341e2a9fb7e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=34 2023-11-27 13:40:16.196 INFO 10 --- [pool-Check host status10,,e7728d7b8d66] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2023-11-27 13:40:16.201 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:40:16.211 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:40:16.218 INFO 10 --- [pool-Check host status7,,6bba02d8f7f0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=38 2023-11-27 13:40:16.238 INFO 10 --- [pool-Check host status4,,8fdf4cc5f47b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-11-27 13:40:16.244 INFO 10 --- [pool-Check host status10,,e7728d7b8d66] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2023-11-27 13:40:16.247 INFO 10 --- [pool-Check host status7,,6bba02d8f7f0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=38 2023-11-27 13:40:16.264 INFO 10 --- [pool-Check host status5,,5341e2a9fb7e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-11-27 13:40:16.284 INFO 10 --- [pool-Check host status4,,8fdf4cc5f47b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=32 2023-11-27 13:40:16.292 INFO 10 --- [pool-Check host status7,,6bba02d8f7f0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-11-27 13:40:16.306 INFO 10 --- [pool-Check host status4,,8fdf4cc5f47b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=32 2023-11-27 13:40:16.311 INFO 10 --- [pool-Check host status5,,5341e2a9fb7e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=35 2023-11-27 13:40:16.354 INFO 10 --- [pool-Check host status7,,6bba02d8f7f0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=39 2023-11-27 13:40:16.358 INFO 10 --- [pool-Check host status5,,5341e2a9fb7e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=35 2023-11-27 13:40:16.363 INFO 10 --- [pool-Check host status10,,e7728d7b8d66] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-11-27 13:40:16.403 INFO 10 --- [pool-Check host status7,,6bba02d8f7f0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=39 2023-11-27 13:40:16.416 INFO 10 --- [pool-Check host status4,,8fdf4cc5f47b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-11-27 13:40:16.419 INFO 10 --- [pool-Check host status10,,e7728d7b8d66] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2023-11-27 13:40:16.431 INFO 10 --- [pool-Check host status5,,5341e2a9fb7e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-11-27 13:40:16.443 INFO 10 --- [pool-Check host status10,,e7728d7b8d66] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2023-11-27 13:40:16.455 INFO 10 --- [pool-Check host status4,,8fdf4cc5f47b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=33 2023-11-27 13:40:16.468 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:40:16.472 INFO 10 --- [pool-Check host status5,,5341e2a9fb7e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=36 2023-11-27 13:40:16.478 INFO 10 --- [pool-Check host status4,,8fdf4cc5f47b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=33 2023-11-27 13:40:16.480 INFO 10 --- [pool-Check host status10,,e7728d7b8d66] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-11-27 13:40:16.492 INFO 10 --- [pool-Check host status5,,5341e2a9fb7e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=36 2023-11-27 13:40:16.546 INFO 10 --- [pool-Check host status10,,e7728d7b8d66] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2023-11-27 13:40:16.582 INFO 10 --- [pool-Check host status10,,e7728d7b8d66] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2023-11-27 13:40:16.603 INFO 10 --- [pool-Check host status15,,9f93e388b2f7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=133.197.206.97, originalStatus=OFFLINE 2023-11-27 13:40:16.609 INFO 10 --- [pool-Check host status15,,9f93e388b2f7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:40:16.611 INFO 10 --- [pool-Check host status11,,66a3fc05b367] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=133.197.204.3, originalStatus=OFFLINE 2023-11-27 13:40:16.611 INFO 10 --- [pool-Check host status9,,b5f3b60522a6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=133.197.206.92, originalStatus=OFFLINE 2023-11-27 13:40:16.611 INFO 10 --- [pool-Check host status16,,8bd51a94efdc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=133.197.204.10, originalStatus=ONLINE 2023-11-27 13:40:16.611 INFO 10 --- [pool-Check host status19,,05cc7cc45ddc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=133.197.204.11, originalStatus=ONLINE 2023-11-27 13:40:16.611 INFO 10 --- [pool-Check host status14,,44227fc78650] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=6, ip=133.197.204.12, originalStatus=ONLINE 2023-11-27 13:40:16.611 INFO 10 --- [pool-Check host status18,,1b4fbfa88258] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=133.197.204.13, originalStatus=ONLINE 2023-11-27 13:40:16.612 INFO 10 --- [pool-Check host status24,,6a2145aba3cf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=9, ip=133.197.206.88, originalStatus=OFFLINE 2023-11-27 13:40:16.632 INFO 10 --- [pool-Check host status24,,6a2145aba3cf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:40:16.635 INFO 10 --- [pool-Check host status11,,66a3fc05b367] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:40:16.636 INFO 10 --- [pool-Check host status18,,1b4fbfa88258] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2023-11-27 13:40:16.637 INFO 10 --- [pool-Check host status9,,b5f3b60522a6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:40:16.637 INFO 10 --- [pool-Check host status16,,8bd51a94efdc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-11-27 13:40:16.638 INFO 10 --- [pool-Check host status19,,05cc7cc45ddc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-11-27 13:40:16.645 INFO 10 --- [pool-Check host status14,,44227fc78650] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2023-11-27 13:40:16.688 INFO 10 --- [pool-Check host status16,,8bd51a94efdc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.10:62888/api/v1/agent/status, params:null 2023-11-27 13:40:16.690 INFO 10 --- [pool-Check host status24,,6a2145aba3cf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.88:62888/api/v1/agent/status, params:null 2023-11-27 13:40:16.692 WARN 10 --- [pool-Check host status24,,6a2145aba3cf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent status check failed. host=9, ip=133.197.206.88, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.88:62888 [/133.197.206.88] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.agentStatus(AgentExecutor.java:622) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostAgentStatus(HostCheckManagerImpl.java:83) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.88:62888 [/133.197.206.88] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 23 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 32 common frames omitted 2023-11-27 13:40:16.696 INFO 10 --- [pool-Check host status24,,6a2145aba3cf] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [9:133.197.206.88], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:40:16.696 INFO 10 --- [pool-Check host status24,,6a2145aba3cf] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:40:16.698 INFO 10 --- [pool-Check host status16,,8bd51a94efdc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=133.197.204.10 2023-11-27 13:40:16.698 INFO 10 --- [pool-Check host status16,,8bd51a94efdc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=133.197.204.10 2023-11-27 13:40:16.698 INFO 10 --- [pool-Check host status16,,8bd51a94efdc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=133.197.204.10 2023-11-27 13:40:16.699 INFO 10 --- [pool-Check host status9,,b5f3b60522a6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.92:62888/api/v1/agent/status, params:null 2023-11-27 13:40:16.701 WARN 10 --- [pool-Check host status9,,b5f3b60522a6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent status check failed. host=3, ip=133.197.206.92, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.92:62888 [/133.197.206.92] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.agentStatus(AgentExecutor.java:622) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostAgentStatus(HostCheckManagerImpl.java:83) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.92:62888 [/133.197.206.92] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 23 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 32 common frames omitted 2023-11-27 13:40:16.705 INFO 10 --- [pool-Check host status9,,b5f3b60522a6] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [3:133.197.206.92], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:40:16.706 INFO 10 --- [pool-Check host status9,,b5f3b60522a6] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:40:16.706 INFO 10 --- [pool-Check host status24,,6a2145aba3cf] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=9 2023-11-27 13:40:16.706 INFO 10 --- [pool-Check host status24,,6a2145aba3cf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=9 2023-11-27 13:40:16.711 INFO 10 --- [pool-Check host status11,,66a3fc05b367] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.3:62888/api/v1/agent/status, params:null 2023-11-27 13:40:16.711 INFO 10 --- [pool-Check host status24,,6a2145aba3cf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:40:16.711 INFO 10 --- [pool-Check host status9,,b5f3b60522a6] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=3 2023-11-27 13:40:16.711 INFO 10 --- [pool-Check host status9,,b5f3b60522a6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2023-11-27 13:40:16.713 WARN 10 --- [pool-Check host status11,,66a3fc05b367] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent status check failed. host=2, ip=133.197.204.3, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.204.3:62888 [/133.197.204.3] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.agentStatus(AgentExecutor.java:622) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostAgentStatus(HostCheckManagerImpl.java:83) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.204.3:62888 [/133.197.204.3] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 23 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 32 common frames omitted 2023-11-27 13:40:16.716 INFO 10 --- [pool-Check host status9,,b5f3b60522a6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:40:16.719 INFO 10 --- [pool-Check host status19,,05cc7cc45ddc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.11:62888/api/v1/agent/status, params:null 2023-11-27 13:40:16.721 INFO 10 --- [pool-Check host status15,,9f93e388b2f7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.97:62888/api/v1/agent/status, params:null 2023-11-27 13:40:16.722 WARN 10 --- [pool-Check host status15,,9f93e388b2f7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent status check failed. host=1, ip=133.197.206.97, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.97:62888 [/133.197.206.97] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.agentStatus(AgentExecutor.java:622) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostAgentStatus(HostCheckManagerImpl.java:83) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.97:62888 [/133.197.206.97] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 23 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 32 common frames omitted 2023-11-27 13:40:16.723 INFO 10 --- [pool-Check host status11,,66a3fc05b367] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [2:133.197.204.3], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:40:16.723 INFO 10 --- [pool-Check host status11,,66a3fc05b367] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:40:16.725 INFO 10 --- [pool-Check host status19,,05cc7cc45ddc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=133.197.204.11 2023-11-27 13:40:16.725 INFO 10 --- [pool-Check host status19,,05cc7cc45ddc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=133.197.204.11 2023-11-27 13:40:16.725 INFO 10 --- [pool-Check host status19,,05cc7cc45ddc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=133.197.204.11 2023-11-27 13:40:16.726 INFO 10 --- [pool-Check host status18,,1b4fbfa88258] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.13:62888/api/v1/agent/status, params:null 2023-11-27 13:40:16.728 INFO 10 --- [pool-Check host status11,,66a3fc05b367] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=2 2023-11-27 13:40:16.728 INFO 10 --- [pool-Check host status11,,66a3fc05b367] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2023-11-27 13:40:16.730 INFO 10 --- [pool-Check host status18,,1b4fbfa88258] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=133.197.204.13 2023-11-27 13:40:16.731 INFO 10 --- [pool-Check host status18,,1b4fbfa88258] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=133.197.204.13 2023-11-27 13:40:16.731 INFO 10 --- [pool-Check host status18,,1b4fbfa88258] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=133.197.204.13 2023-11-27 13:40:16.731 INFO 10 --- [pool-Check host status15,,9f93e388b2f7] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [1:133.197.206.97], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:40:16.731 INFO 10 --- [pool-Check host status15,,9f93e388b2f7] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:40:16.734 INFO 10 --- [pool-Check host status11,,66a3fc05b367] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:40:16.737 INFO 10 --- [pool-Check host status15,,9f93e388b2f7] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=1 2023-11-27 13:40:16.737 INFO 10 --- [pool-Check host status15,,9f93e388b2f7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2023-11-27 13:40:16.738 INFO 10 --- [pool-Check host status14,,44227fc78650] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.12:62888/api/v1/agent/status, params:null 2023-11-27 13:40:16.739 INFO 10 --- [pool-Check host status15,,9f93e388b2f7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:40:16.743 INFO 10 --- [pool-Check host status14,,44227fc78650] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=133.197.204.12 2023-11-27 13:40:16.743 INFO 10 --- [pool-Check host status14,,44227fc78650] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=133.197.204.12 2023-11-27 13:40:16.743 INFO 10 --- [pool-Check host status14,,44227fc78650] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=133.197.204.12 2023-11-27 13:40:17.341 INFO 10 --- [http-nio-0.0.0.0-8080-exec-7,b93f90d07f774878,98e7ecd189dd] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.11, traceId=b93f90d07f774878, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:40:17.430 INFO 10 --- [http-nio-0.0.0.0-8080-exec-7,b93f90d07f774878,98e7ecd189dd] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.11, traceId=b93f90d07f774878, duration=90 ms] 2023-11-27 13:40:19.212 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:40:19.213 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:40:20.131 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063620, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:20.131 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063620, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:20.132 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063620, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:20.134 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063620, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:20.131 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063620, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:20.881 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:39:20.0, nextRun=2023-11-27 13:40:20.0, currentDate=Mon Nov 27 13:40:20 CST 2023 2023-11-27 13:40:20.882 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-11-27 13:39:20.0, nextRunTime=2023-11-27 13:40:20.0} 2023-11-27 13:40:20.910 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-11-27 13:40:20.911 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:40:20.911 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-11-27 13:40:21.087 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000089, operation:EXECUTE, state: RUNNING 2023-11-27 13:40:21.092 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000089, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-11-27T13:40:21.079+08:00, endTime=null} 2023-11-27 13:40:21.147 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,2d9ca292b636] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000089, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000075, task_operation=execute, latest_execution_start_time=2023-11-27T13:40:21.079+08:00, sub_task_instance_id=10000089}, listMap={}}, executor=10.120.0.1 2023-11-27 13:40:21.163 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,2d9ca292b636] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-11-27 13:40:21.164 INFO 10 --- [ForkJoinPool-44-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-11-27 13:40:21.170 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,2d9ca292b636] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000089, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:40:21.289 INFO 10 --- [ForkJoinPool-44-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:21.289 INFO 10 --- [ForkJoinPool-44-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:21.290 INFO 10 --- [ForkJoinPool-44-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:21.290 INFO 10 --- [ForkJoinPool-44-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:21.291 INFO 10 --- [ForkJoinPool-44-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:21.291 INFO 10 --- [ForkJoinPool-44-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:21.291 INFO 10 --- [ForkJoinPool-44-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT MAX(CASE `name` WHEN 'cluster' THEN `info` ELSE '' END ) `cluster`, MAX(CASE `name` WHEN 'merge_status' THEN `info` ELSE '' END ) `merge_status`, MAX(CASE `name` WHEN 'global_broadcast_version' THEN `value` ELSE 0 END ) `global_broadcast_version`, MAX(CASE `name` WHEN 'frozen_version' THEN `value` ELSE 0 END ) `frozen_version`, MAX(CASE `name` WHEN 'last_merged_version' THEN `value` ELSE 0 END ) `last_merged_version`, MAX(CASE `name` WHEN 'frozen_time' THEN `value` ELSE 0 END ) `frozen_time`, MAX(CASE `name` WHEN 'is_merge_error' THEN `value` ELSE 0 END ) `merge_error` FROM oceanbase.__all_zone WHERE `zone` = '' 2023-11-27 13:40:21.293 INFO 10 --- [ForkJoinPool-44-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:21.294 INFO 10 --- [ForkJoinPool-44-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:21.294 INFO 10 --- [ForkJoinPool-44-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select column_value from oceanbase.__all_core_table where table_name ='__all_global_stat' and column_name = 'frozen_version' 2023-11-27 13:40:21.295 INFO 10 --- [ForkJoinPool-44-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:21.295 INFO 10 --- [ForkJoinPool-44-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:21.295 INFO 10 --- [ForkJoinPool-44-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, MAX(CASE `name` WHEN 'region' THEN `info` ELSE '' END ) `region`, MAX(CASE `name` WHEN 'idc' THEN `info` ELSE '' END ) `idc`, MAX(CASE `name` WHEN 'status' THEN `info` ELSE '' END ) `status`, MAX(CASE `name` WHEN 'merge_status' THEN `info` ELSE '' END ) `merge_status`, MAX(CASE `name` WHEN 'broadcast_version' THEN `value` ELSE 0 END ) `broadcast_version`, MAX(CASE `name` WHEN 'all_merged_version' THEN `value` ELSE 0 END ) `all_merged_version`, MAX(CASE `name` WHEN 'last_merged_version' THEN `value` ELSE 0 END ) `last_merged_version`, MAX(CASE `name` WHEN 'merge_start_time' THEN `value` ELSE 0 END ) `merge_start_time`, MAX(CASE `name` WHEN 'last_merged_time' THEN `value` ELSE 0 END ) `last_merged_time`, MAX(CASE `name` WHEN 'is_merge_timeout' THEN `value` ELSE 0 END ) `merge_timeout` FROM oceanbase.__all_zone WHERE `zone` <> '' GROUP BY `zone` 2023-11-27 13:40:21.302 INFO 10 --- [ForkJoinPool-44-worker-2,,] s.t.b.c.CollectAllClustersCompactionTask : compaction collect success, cluster=ObClusterOverview(id=1, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"133.197.206.97:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1701063583198000}, createTime=2022-11-29T14:52:53+08:00, creator=admin, obVersion=3.1.4, tenantCount=3) 2023-11-27 13:40:21.331 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000075, status:SUCCESS 2023-11-27 13:40:21.339 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000075, status:SUCCESS 2023-11-27 13:40:21.816 INFO 10 --- [pool-overview-container-8,,83af3b29fb4d] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=420, alarmHistory size=15892 2023-11-27 13:40:21.924 INFO 10 --- [pool-scanner--12639-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=76 2023-11-27 13:40:22.213 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:40:22.215 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:40:22.779 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:40:22.778142+08:00, lastProcessTime=2023-11-27T13:40:12.789260+08:00 2023-11-27 13:40:22.903 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:40:22.778142+08:00 2023-11-27 13:40:23.557 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.120.0.1:8080 2023-11-27 13:40:23.683 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.120.0.1:8080 2023-11-27 13:40:23.933 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.120.0.1:8080 2023-11-27 13:40:24.235 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:40:24.486 INFO 10 --- [alarm-detect-10,,f357ef12481c] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_cluster_exists_inactive_server detect time range 1701063604 ~ 1701063614 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={inactive_server_count > 0=[ExpressionEvaluateResult(matchValue=0.0, metric=inactive_server_count, match=true, value=1.0, values={1701063610=1.0}, valueCount=1, labels={app=OB, ob_cluster_id=10, ob_cluster_name=darenOB, server_ips=133.197.204.3})]}, matched=true, message=null, level=2) 2023-11-27 13:40:24.486 INFO 10 --- [alarm-detect-10,,f357ef12481c] c.o.o.s.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_cluster_exists_inactive_server, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=darenOB-10-2] 2023-11-27 13:40:24.741 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.120.0.1:8080 2023-11-27 13:40:24.869 INFO 10 --- [pool-ocp-schedules-6,,b3355cc20aae] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:40:25.102 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063625, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:25.102 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063625, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:25.103 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063625, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:25.104 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063625, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:25.105 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063625, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:25.143 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:39:25.0, nextRun=2023-11-27 13:40:25.0, currentDate=Mon Nov 27 13:40:25 CST 2023 2023-11-27 13:40:25.143 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-11-27 13:39:25.0, nextRunTime=2023-11-27 13:40:25.0} 2023-11-27 13:40:25.185 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-11-27 13:40:25.185 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:40:25.185 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-11-27 13:40:25.215 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:40:25.216 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:40:25.434 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:40:25.445 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:40:25.478 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:40:25.807 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:40:26.174 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000090, operation:EXECUTE, state: RUNNING 2023-11-27 13:40:26.180 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000090, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-11-27T13:40:26.171+08:00, endTime=null} 2023-11-27 13:40:26.248 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,21f4bfacd944] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000090, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000076, task_operation=execute, latest_execution_start_time=2023-11-27T13:40:26.170+08:00, sub_task_instance_id=10000090}, listMap={host_ids=[1, 2, 3, 4, 5, 6, 7, 9]}}, executor=10.120.0.1 2023-11-27 13:40:26.345 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,21f4bfacd944] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=271051, maxInactiveSeconds=180 2023-11-27 13:40:26.345 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,21f4bfacd944] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=271051, maxInactiveSeconds=180 2023-11-27 13:40:26.345 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,21f4bfacd944] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=271051, maxInactiveSeconds=180 2023-11-27 13:40:26.345 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,21f4bfacd944] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=271051, maxInactiveSeconds=180 2023-11-27 13:40:26.345 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,21f4bfacd944] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=271051, maxInactiveSeconds=180 2023-11-27 13:40:26.345 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,21f4bfacd944] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=271051, maxInactiveSeconds=180 2023-11-27 13:40:26.345 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,21f4bfacd944] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=271051, maxInactiveSeconds=180 2023-11-27 13:40:26.345 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,21f4bfacd944] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=271051, maxInactiveSeconds=180 2023-11-27 13:40:26.345 WARN 10 --- [pool-subtask-executor32,554481c3fd1b426b,21f4bfacd944] c.o.o.s.t.b.host.CheckExporterTask : Host [id=1, ip=133.197.206.97, status=OFFLINE] not online, skip check exporter 2023-11-27 13:40:26.345 WARN 10 --- [pool-subtask-executor32,554481c3fd1b426b,21f4bfacd944] c.o.o.s.t.b.host.CheckExporterTask : Host [id=2, ip=133.197.204.3, status=OFFLINE] not online, skip check exporter 2023-11-27 13:40:26.345 WARN 10 --- [pool-subtask-executor32,554481c3fd1b426b,21f4bfacd944] c.o.o.s.t.b.host.CheckExporterTask : Host [id=3, ip=133.197.206.92, status=OFFLINE] not online, skip check exporter 2023-11-27 13:40:26.347 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,21f4bfacd944] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=133.197.204.10] desired exporter count: 8, actual exporter count: 8 2023-11-27 13:40:26.348 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,21f4bfacd944] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=133.197.204.11] desired exporter count: 8, actual exporter count: 8 2023-11-27 13:40:26.349 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,21f4bfacd944] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=6, ip=133.197.204.12] desired exporter count: 8, actual exporter count: 8 2023-11-27 13:40:26.350 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,21f4bfacd944] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=7, ip=133.197.204.13] desired exporter count: 8, actual exporter count: 8 2023-11-27 13:40:26.350 WARN 10 --- [pool-subtask-executor32,554481c3fd1b426b,21f4bfacd944] c.o.o.s.t.b.host.CheckExporterTask : Host [id=9, ip=133.197.206.88, status=OFFLINE] not online, skip check exporter 2023-11-27 13:40:26.375 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,21f4bfacd944] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000090, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:40:26.495 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:40:26.541 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000076, status:SUCCESS 2023-11-27 13:40:26.542 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000076, status:SUCCESS 2023-11-27 13:40:27.942 INFO 10 --- [pool-ocp-schedules-4,,037239bb1285] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-11-27 13:40:28.152 INFO 10 --- [pool-ocp-schedules-4,,037239bb1285] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-11-27 13:40:28.216 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:40:28.227 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:40:28.465 INFO 10 --- [http-nio-0.0.0.0-8080-exec-10,403059c5eb3f451e,8688f8cc92cb] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.11, traceId=403059c5eb3f451e, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:40:28.529 INFO 10 --- [http-nio-0.0.0.0-8080-exec-10,403059c5eb3f451e,8688f8cc92cb] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.11, traceId=403059c5eb3f451e, duration=65 ms] 2023-11-27 13:40:30.111 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063630, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:30.111 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063630, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:30.111 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063630, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:30.112 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063630, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:30.115 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063630, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:30.476 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:39:31.0, nextRun=2023-11-27 13:40:30.0, currentDate=Mon Nov 27 13:40:30 CST 2023 2023-11-27 13:40:30.476 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-11-27 13:39:31.0, nextRunTime=2023-11-27 13:40:30.0} 2023-11-27 13:40:30.519 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-11-27 13:40:30.519 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:40:30.519 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:40:30.519 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-11-27 13:40:30.533 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:40:01.0, nextRun=2023-11-27 13:40:30.0, currentDate=Mon Nov 27 13:40:30 CST 2023 2023-11-27 13:40:30.534 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-11-27 13:40:01.0, nextRunTime=2023-11-27 13:40:30.0} 2023-11-27 13:40:30.561 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-11-27 13:40:30.561 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:40:30.561 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-11-27 13:40:31.227 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:40:31.228 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:40:31.284 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000091, operation:EXECUTE, state: RUNNING 2023-11-27 13:40:31.286 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000091, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-11-27T13:40:31.281+08:00, endTime=null} 2023-11-27 13:40:31.303 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000092, operation:EXECUTE, state: RUNNING 2023-11-27 13:40:31.306 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000092, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-11-27T13:40:31.300+08:00, endTime=null} 2023-11-27 13:40:31.323 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000093, operation:EXECUTE, state: RUNNING 2023-11-27 13:40:31.330 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000093, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-11-27T13:40:31.320+08:00, endTime=null} 2023-11-27 13:40:31.340 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000091, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=8000077, task_operation=execute, latest_execution_start_time=2023-11-27T13:40:31.281+08:00, sub_task_instance_id=10000091}, listMap={cluster_ids=[1, 2]}}, executor=10.120.0.1 2023-11-27 13:40:31.355 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000092, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=8000077, task_operation=execute, latest_execution_start_time=2023-11-27T13:40:31.299+08:00, sub_task_instance_id=10000092}, listMap={cluster_ids=[1, 2]}}, executor=10.120.0.1 2023-11-27 13:40:31.362 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,134535e8fea3] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000093, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000078, task_operation=execute, latest_execution_start_time=2023-11-27T13:40:31.320+08:00, sub_task_instance_id=10000093}, listMap={}}, executor=10.120.0.1 2023-11-27 13:40:31.362 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,134535e8fea3] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-11-27 13:40:31.427 INFO 10 --- [pool-1413-thread-1,554481c3fd1b426b,46011604cd93] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:darenOB, tenant:sys 2023-11-27 13:40:31.431 INFO 10 --- [pool-1413-thread-2,554481c3fd1b426b,0f5fa5ac968a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:darenOB, tenant:daren 2023-11-27 13:40:31.531 INFO 10 --- [pool-1413-thread-2,554481c3fd1b426b,0f5fa5ac968a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:31.532 INFO 10 --- [pool-1413-thread-2,554481c3fd1b426b,0f5fa5ac968a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:31.532 INFO 10 --- [pool-1413-thread-2,554481c3fd1b426b,0f5fa5ac968a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.534 INFO 10 --- [pool-1413-thread-2,554481c3fd1b426b,0f5fa5ac968a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:31.535 INFO 10 --- [pool-1413-thread-1,554481c3fd1b426b,46011604cd93] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:31.537 INFO 10 --- [pool-1413-thread-1,554481c3fd1b426b,46011604cd93] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:31.537 INFO 10 --- [pool-1413-thread-1,554481c3fd1b426b,46011604cd93] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.538 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.538 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.539 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.539 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:31.540 INFO 10 --- [pool-1413-thread-1,554481c3fd1b426b,46011604cd93] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:31.540 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:31.541 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.541 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.541 INFO 10 --- [pool-1413-thread-2,554481c3fd1b426b,0f5fa5ac968a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:31.541 INFO 10 --- [pool-1413-thread-2,554481c3fd1b426b,0f5fa5ac968a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.541 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, tenant_name, (CASE WHEN tenant_id = 1 THEN 'SYS' ELSE 'USER' END) AS tenant_type, (CASE WHEN compatibility_mode = 1 THEN 'ORACLE' ELSE 'MYSQL' END) AS compatibility_mode, zone_list, primary_zone, locality, status, locked, read_only, in_recyclebin, TIMESTAMPDIFF(SECOND, gmt_create, now()) AS exist_seconds FROM oceanbase.__all_tenant 2023-11-27 13:40:31.542 INFO 10 --- [pool-1413-thread-2,554481c3fd1b426b,0f5fa5ac968a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-11-27 13:40:31.546 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.546 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.547 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1] 2023-11-27 13:40:31.548 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:31.548 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.549 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:31.552 INFO 10 --- [pool-1413-thread-1,554481c3fd1b426b,46011604cd93] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:31.552 INFO 10 --- [pool-1413-thread-1,554481c3fd1b426b,46011604cd93] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.559 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:31.559 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.559 INFO 10 --- [pool-1413-thread-1,554481c3fd1b426b,46011604cd93] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-11-27 13:40:31.560 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-11-27 13:40:31.560 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.560 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.561 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1] 2023-11-27 13:40:31.568 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:31.568 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.569 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-11-27 13:40:31.573 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.573 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.573 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1001] 2023-11-27 13:40:31.581 INFO 10 --- [pool-1413-thread-2,554481c3fd1b426b,0f5fa5ac968a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1701021600396785720, broadcast scn:1701021600396785720, saved frozen scn:1701021600396785720 2023-11-27 13:40:31.581 INFO 10 --- [pool-1413-thread-2,554481c3fd1b426b,0f5fa5ac968a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:darenOB, tenant:daren 2023-11-27 13:40:31.585 INFO 10 --- [pool-1413-thread-1,554481c3fd1b426b,46011604cd93] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1701021601748952808, broadcast scn:1701021601748952808, saved frozen scn:1701021601748952808 2023-11-27 13:40:31.586 INFO 10 --- [pool-1413-thread-1,554481c3fd1b426b,46011604cd93] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:darenOB, tenant:sys 2023-11-27 13:40:31.586 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:31.586 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.587 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-11-27 13:40:31.597 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.597 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.598 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,134535e8fea3] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-11-27 13:40:31.598 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1001] 2023-11-27 13:40:31.603 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:31.604 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.604 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-11-27 13:40:31.606 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,134535e8fea3] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000093, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:40:31.618 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.618 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.619 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:31.619 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.620 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1002] 2023-11-27 13:40:31.620 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-11-27 13:40:31.632 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-11-27 13:40:31.638 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.638 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.638 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1002] 2023-11-27 13:40:31.648 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-11-27 13:40:31.688 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:40:31.689 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.691 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.691 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.691 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:31.693 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.693 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.693 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-11-27 13:40:31.714 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:40:31.714 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.715 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.715 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.715 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:31.717 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.717 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.717 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:40:31.728 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:40:31.729 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:31.730 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:31.730 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.730 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:31.734 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:31.734 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.735 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-11-27 13:40:31.749 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:40:31.750 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.751 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.751 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.753 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:31.754 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.754 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.755 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1001, ALTER_TENANT_LOCALITY] 2023-11-27 13:40:31.772 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000078, status:SUCCESS 2023-11-27 13:40:31.774 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000078, status:SUCCESS 2023-11-27 13:40:31.775 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:40:31.775 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.776 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.776 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.776 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:31.778 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.778 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.779 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1001, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:40:31.809 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:40:31.810 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.810 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.810 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.810 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:31.813 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.813 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.813 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-11-27 13:40:31.813 INFO 10 --- [pool-overview-container-1,,90f5d02750f9] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=420, alarmHistory size=15892 2023-11-27 13:40:31.820 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:40:31.821 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:31.822 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:31.822 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.822 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:31.827 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:40:31.828 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.829 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.829 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.829 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:31.829 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:31.829 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.830 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.830 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.831 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:40:31.831 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:40:31.831 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-11-27 13:40:31.867 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.120.0.1 2023-11-27 13:40:31.882 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:40:31.883 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:31.884 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:31.884 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.885 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:31.888 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:40:31.889 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:31.889 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.890 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.891 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.891 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.891 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:31.895 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-11-27 13:40:31.897 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.897 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.897 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1] 2023-11-27 13:40:31.909 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:40:31.916 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-11-27 13:40:31.917 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.918 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.919 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1001] 2023-11-27 13:40:31.922 INFO 10 --- [pool-scanner--12732-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=76 2023-11-27 13:40:31.930 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:40:31.937 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:40:31.938 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:31.940 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:31.940 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.940 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-11-27 13:40:31.941 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:31.941 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:31.941 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.942 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1002] 2023-11-27 13:40:31.948 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:31.948 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:31.949 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:40:31.950 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-11-27 13:40:31.951 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:40:31.955 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-11-27 13:40:31.955 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-11-27 13:40:31.955 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=obcluster, obClusterId=1 2023-11-27 13:40:31.956 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.120.0.1 2023-11-27 13:40:31.973 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,dd3d0576b562] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000091, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:40:32.027 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:40:32.029 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:32.029 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:32.030 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:32.030 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:32.070 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:32.070 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:32.071 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-11-27 13:40:32.082 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:40:32.086 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-11-27 13:40:32.087 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:32.087 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:32.088 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-11-27 13:40:32.105 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:40:32.112 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-11-27 13:40:32.112 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-11-27 13:40:32.112 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=darenOB, obClusterId=10 2023-11-27 13:40:32.128 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,9a0e9916930d] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000092, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:40:32.787 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:40:32.786581+08:00, lastProcessTime=2023-11-27T13:40:22.778142+08:00 2023-11-27 13:40:32.847 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000077, status:SUCCESS 2023-11-27 13:40:32.849 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000077, status:SUCCESS 2023-11-27 13:40:32.919 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : groupAggregate, now=2023-11-27T13:40:32.786581+08:00, groupByKey=0:host_unavailable:1, alarmsToSentCount=4, alarmIds=[44800, 2000006, 2000007, 2000005] 2023-11-27 13:40:32.919 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : createNotification start, groupByKey=0:host_unavailable:1 2023-11-27 13:40:32.919 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : createNotification done, groupByKey=0:host_unavailable:1, alarmIds=[44800, 2000006, 2000007, 2000005] 2023-11-27 13:40:32.919 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : groupAggregate, now=2023-11-27T13:40:32.786581+08:00, groupByKey=0:monitor_exporter_unavaliable:3, alarmsToSentCount=7, alarmIds=[44801, 44802, 44803, 44804, 44805, 44791, 44796] 2023-11-27 13:40:32.920 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : createNotification start, groupByKey=0:monitor_exporter_unavaliable:3 2023-11-27 13:40:32.920 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : createNotification done, groupByKey=0:monitor_exporter_unavaliable:3, alarmIds=[44801, 44802, 44803, 44804, 44805, 44791, 44796] 2023-11-27 13:40:32.920 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.ChannelAggregator : aggregate done, channelId=100001, now=2023-11-27T13:40:32.786581+08:00, resultsCount=2 2023-11-27 13:40:32.920 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : aggregate done, channelId=100001, alarmsCount=1, resultsCount=2 2023-11-27 13:40:32.920 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.ChannelAggregator : groupAggregate markSendSuccess, groupByKey=0:host_unavailable:1, now=2023-11-27T13:40:32.786581+08:00, alarmIds=[44800, 2000006, 2000007, 2000005] 2023-11-27 13:40:32.920 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.ChannelAggregator : groupAggregate markSendSuccess, groupByKey=0:monitor_exporter_unavaliable:3, now=2023-11-27T13:40:32.786581+08:00, alarmIds=[44801, 44802, 44803, 44804, 44805, 44791, 44796] 2023-11-27 13:40:32.921 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:40:32.786581+08:00 2023-11-27 13:40:33.976 INFO 10 --- [pool-prometheus-cache-monitor1,,a0c915e1cfc9] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":6828,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":1707,"seriesIdKeySize":87133,"secondWriteQueue":0,"minuteWriteCacheSize":32725} 2023-11-27 13:40:33.976 INFO 10 --- [pool-prometheus-cache-monitor1,,a0c915e1cfc9] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_zone_current_timestamp=38, ob_server_resource_memory_assigned_bytes=21, tcp_retrans=30, ob_unit_num=21, go_memstats_mspan_sys_bytes=17, go_memstats_stack_sys_bytes=17, exporter_export_metrics_bytes_total=28, node_filesystem_size_bytes=87, http_server_requests_seconds_count=40, ob_compaction_frozen_time=18, ob_system_event_total_waits=34, monagent_input_sql_duration_seconds_sum=60, observer_fd_count=20, ob_plan_cache_hit_total=28, processor_process_metrics_total=22, node_filesystem_readonly=86, ob_server_resource_disk_in_use_bytes=19, go_memstats_frees_total=17, sql_audit_input_collect_data_jumped_total=12, io_await=60, host_available=13, go_memstats_mallocs_total=17, mysql_output_write_sql_failed_count=23, ob_unit_config_max_memory_bytes=45, io_util=60, ob_system_event_detail_total_timeouts=782, ob_tenant_disk_v4_data_size=18, ob_latch_sleep_total=326, ob_tenant_unit_min_memory=41, ob_memstore_total_bytes=28, process_resident_memory_bytes=17, ob_cache_size_bytes=47, ob_tenant_unit_max_memory=41, ob_tenant_assigned_cpu_total=25, input_collect_metrics_total=29, ob_tenant_server_available_disk_free=28, ob_tenant_server_avg_cpu_used_percentage=23, node_disk_io_time_weighted_seconds_total=39, node_disk_write_time_seconds_total=39, cpu_count=30, output_write_metrics_total=15, ob_cluster_sync=175, process_open_fds=17, odp_transaction_total=22, ob_system_event_detail_time_waited_us=782, process_max_fds=17, ob_server_resource_cpu_assigned_percent=20, ocp_alarm_detect_duration_ms_count=82, go_memstats_heap_inuse_bytes=17, ob_tenant_server_max_cpu_used_percentage=23, proxy_memory_used_bytes=30, go_memstats_mcache_sys_bytes=17, ob_tenant_disk_v4_log_size=18, node_memory_memfree_bytes=29, process_exists=38, go_gc_duration_seconds=22, ob_server_resource_disk_bytes=21, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=27, node_memory_cached_bytes=29, ob_table_num=24, ob_disk_free_bytes=21, node_filesystem_avail_bytes=87, ob_backup_storage_capacity_bytes=349, ob_active_session_num=27, ob_unit_config_max_iops=45, ob_plan_cache_memory_bytes=28, node_network_transmit_errs_total=65, go_memstats_last_gc_time_seconds=17, go_gc_duration_seconds_count=17, monitor_exporter_avaliable=78, ob_tenant_server_max_memory_used_percentage=23, node_disk_read_bytes_total=39, mysql_output_write_sql_milliseconds_summary_count=25, ob_compaction_is_suspended=18, ob_tenant_unit_log_disk_size=22, go_memstats_lookups_total=17, mysql_output_write_sql_count=25, log_tailer_count=17, host_agent_process_status=21, ob_server_resource_cpu=21, go_memstats_alloc_bytes=17, ob_server_num=58, node_cpu_seconds_total_native=47, ob_tenant_unit_count=39, ob_session_all_num=25, ob_tenant_resource_min_memory=25, ob_memstore_freeze_times=28, ob_system_event_time_waited=34, ob_tenant_resource_min_cpu=25, ob_cluster_status_check=49, monagent_plugin_execute_seconds_total=41, node_net_bandwidth_bps=45, monagent_oceanbase_db_pool_idle=7, go_memstats_heap_idle_bytes=17, go_memstats_gc_sys_bytes=17, ob_server_resource_memory_bytes=21, ob_tenant_sys_partition_required_size=12, ob_latch_get_total=326, go_info=18, ob_tenant_assigned_mem_assigned=25, log_tailer_processed_file_count=24, node_load15=29, ob_tenant_server_required_size=35, ob_unit_config_min_iops=45, ob_server_partition_replica_limit=7, ob_tenant_unit_max_cpu=41, ob_latch_wait_seconds_total=326, node_memory_buffers_bytes=29, go_memstats_heap_released_bytes=17, node_network_receive_bytes_total=65, ob_memstore_freeze_trigger_bytes=28, ob_waitevent_wait_seconds_total=28, ob_server_stopped_duration_seconds=20, exposer_request_latencies_count=8, ob_role_is_rootserver=20, go_memstats_next_gc_bytes=17, mysql_output_context_timeout_discard_metrics_count=17, ob_session_active_num=25, node_filesystem_files_free=109, ob_memtable_max_snapshot_duration_seconds=27, node_disk_read_time_seconds_total=39, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=22, process_virtual_memory_bytes=17, mysql_output_write_sql_retry_count=23, ob_tenant_assigned_cpu_assigned=25, ob_tenant_unit_min_cpu=41, ob_unit_config_max_cpu=45, ob_unit_config_max_session_num=18, http_request_milliseconds_summary_count=79, monagent_oceanbase_audit_timeinfo_cache_size=7, net_connectivity=10, monagent_input_sql_duration_seconds_bucket=70, ob_latch_immediate_get_total=326, node_cpu_seconds_total=37, http_request_milliseconds_summary=82, ob_latch_immediate_miss_total=326, ob_index_error_num=16, ob_tenant_server_available_disk_total=28, ob_latch_spin_get_total=326, ob_tenant_memory_hold_bytes=18, go_memstats_buck_hash_sys_bytes=17, monagent_pipeline_report_metrics_total=18, mysql_output_write_sql_milliseconds_summary_sum=25, ob_tenant_disk_total_size=25, exposer_transferred_bytes_total=8, odp_sql_request_total=67, mysql_output_write_task_size=24, ob_tenant_context_memory_count=339, node_disk_reads_completed_total=39, exporter_export_metrics_total=28, ob_sysstat=129, ob_partition_num=13, node_ntp_offset_seconds=29, ob_latch_miss_total=326, go_memstats_heap_sys_bytes=17, ocp_meta_db_health=4, ob_disk_total_bytes=21, ob_zone_stat=38, go_memstats_heap_objects=17, go_memstats_mcache_inuse_bytes=17, go_memstats_other_sys_bytes=17, go_threads=17, http_request_milliseconds_summary_sum=79, ob_memstore_active_bytes=28, http_server_requests_seconds_sum=40, go_memstats_alloc_bytes_total=17, ob_tenant_server_partition_replica_count=35, ob_tenant_assigned_mem_total=25, node_memory_memtotal_bytes=29, node_network_transmit_drop_total=65, ob_system_event_detail_total_waits=782, odp_used_connections=28, ob_tenant_task_max_duration_seconds=25, backup_storage_capacity_used_percent=14, exposer_request_latencies=11, ob_waitevent_wait_total=28, odp_sql_cost_total=66, ob_tenant500_memory_used_bytes=20, node_network_transmit_bytes_total=65, ob_compaction_is_error=18, node_network_receive_drop_total=65, ob_server_partition_replica_count=7, monagent_pipeline_execute_total=27, node_filesystem_files=109, ob_tenant_server_avg_memory_used_percentage=23, process_start_time_seconds=17, go_memstats_heap_alloc_bytes=17, ob_tenant_resource_max_cpu=25, go_gc_duration_seconds_sum=17, go_memstats_mspan_inuse_bytes=17, mysql_output_write_sql_milliseconds_summary=28, ob_server_resource_memory_assigned_percent=20, monagent_pipeline_execute_seconds_total=27, go_memstats_sys_bytes=17, node_network_receive_errs_total=65, ob_all_session_num=27, observer_ulimit_max_fd_count=20, ob_plan_cache_access_total=28, ob_tenant_server_row_count=35, ob_tenant_memory_used_bytes=18, monagent_oceanbase_plan_cache_size=7, ob_tenant_server_data_size=35, go_memstats_gc_cpu_fraction=17, monagent_oceanbase_audit_cache_size=7, ob_tenant_context_memory_used_bytes=339, node_disk_writes_completed_total=39, node_load1=29, process_cpu_seconds_total=17, ob_tenant_resource_max_memory=25, go_memstats_stack_inuse_bytes=17, ob_tenant_user_partition_required_size=12, node_load5=29, monagent_input_sql_duration_seconds_count=60, node_disk_written_bytes_total=39, monagent_oceanbase_ash_cache_size=7, ob_tenant_context_memory_hold_bytes=339, go_goroutines=17, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=21, exposer_request_latencies_sum=8, ob_unit_config_min_cpu=45, process_virtual_memory_max_bytes=17, monagent_oceanbase_plan_timeinfo_cache_size=7, monagent_plugin_execute_total=41, ob_tenant500_memory_hold_bytes=20, exposer_scrapes_total=8, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=22, ob_compaction_start_time=18, oceanbase_connectivity=21} 2023-11-27 13:40:34.228 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:40:34.230 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:40:34.235 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:40:34.457 INFO 10 --- [alarm-detect-5,,7de09447b761] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_cluster_exists_inactive_server detect time range 1701063614 ~ 1701063624 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={inactive_server_count > 0=[ExpressionEvaluateResult(matchValue=0.0, metric=inactive_server_count, match=true, value=1.0, values={1701063620=1.0}, valueCount=1, labels={app=OB, ob_cluster_id=10, ob_cluster_name=darenOB, server_ips=133.197.204.3})]}, matched=true, message=null, level=2) 2023-11-27 13:40:34.888 INFO 10 --- [pool-ocp-schedules-9,,0eca4c2bd768] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:40:35.102 INFO 10 --- [pool-prometheus-second-schedule-2,,046c7e211d11] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=14, collectTaskCount=14 2023-11-27 13:40:35.103 INFO 10 --- [pool-prometheus-second-schedule-2,,8f5e0527e352] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=11, collectTaskCount=11 2023-11-27 13:40:35.103 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063635, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:35.103 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063635, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:35.104 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063635, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:35.104 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063635, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:35.104 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063635, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:35.438 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:40:35.456 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:40:35.489 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:40:35.777 INFO 10 --- [http-nio-0.0.0.0-8080-exec-9,242de6541f914942,9b6565f21b36] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.10, traceId=242de6541f914942, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:40:35.816 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:40:35.819 INFO 10 --- [http-nio-0.0.0.0-8080-exec-9,242de6541f914942,9b6565f21b36] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.10, traceId=242de6541f914942, duration=43 ms] 2023-11-27 13:40:36.172 INFO 10 --- [pool-ocp-schedules-10,,a732fd71de79] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:40:36.173 INFO 10 --- [pool-ocp-schedules-10,,a732fd71de79] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:40:36.516 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:40:37.230 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:40:37.231 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:40:37.915 INFO 10 --- [http-nio-0.0.0.0-8080-exec-2,1c2f2aad25dd486c,a8a0315871ab] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.10, traceId=1c2f2aad25dd486c, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:40:37.978 INFO 10 --- [http-nio-0.0.0.0-8080-exec-2,1c2f2aad25dd486c,a8a0315871ab] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.10, traceId=1c2f2aad25dd486c, duration=65 ms] 2023-11-27 13:40:39.752 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.120.0.1:8080 2023-11-27 13:40:40.104 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063640, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:40.104 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063640, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:40.104 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063640, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:40.104 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063640, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:40.105 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063640, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:40.231 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:40:40.233 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:40:41.799 INFO 10 --- [pool-overview-container-5,,4e35106192ef] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=420, alarmHistory size=15892 2023-11-27 13:40:41.916 INFO 10 --- [pool-scanner--12825-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=76 2023-11-27 13:40:42.219 INFO 10 --- [http-nio-0.0.0.0-8080-exec-4,0c13583621d74630,655041774cec] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.12, traceId=0c13583621d74630, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:40:42.252 INFO 10 --- [http-nio-0.0.0.0-8080-exec-4,0c13583621d74630,655041774cec] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.12, traceId=0c13583621d74630, duration=33 ms] 2023-11-27 13:40:42.281 INFO 10 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2023-11-27T13:36:42.006, end=2023-11-27T13:38:42.278} 2023-11-27 13:40:42.318 INFO 10 --- [pool-13-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:36:42.006, end=2023-11-27T13:38:42.278},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-11-27 13:40:42.318 INFO 10 --- [pool-13-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null) 2023-11-27 13:40:42.322 INFO 10 --- [pool-13-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 6 diagnose targets Pair(left=Tenant(id=5, name=daren, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1,zone3,zone4,zone5,zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:36:42.006, end=2023-11-27T13:38:42.278},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-11-27 13:40:42.325 INFO 10 --- [pool-13-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=daren, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1,zone3,zone4,zone5,zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null) 2023-11-27 13:40:42.328 INFO 10 --- [pool-13-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1001, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:36:42.006, end=2023-11-27T13:38:42.278},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-11-27 13:40:42.328 INFO 10 --- [pool-13-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=meta_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1001, sqlAuditEnabled=null), right=null) 2023-11-27 13:40:42.330 INFO 10 --- [pool-13-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1;zone3;zone4;zone5;zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:36:42.006, end=2023-11-27T13:38:42.278},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-11-27 13:40:42.330 INFO 10 --- [pool-13-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1;zone3;zone4;zone5;zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null) 2023-11-27 13:40:42.331 INFO 10 --- [pool-13-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:36:42.006, end=2023-11-27T13:38:42.278},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-11-27 13:40:42.332 INFO 10 --- [pool-13-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1, sqlAuditEnabled=null), right=null) 2023-11-27 13:40:42.332 INFO 10 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2023-11-27 13:40:42.761 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:40:42.760582+08:00, lastProcessTime=2023-11-27T13:40:32.786581+08:00 2023-11-27 13:40:42.845 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:40:42.760582+08:00 2023-11-27 13:40:42.879 INFO 10 --- [http-nio-0.0.0.0-8080-exec-8,27a2c1ccc69f405a,420eeb5c026d] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.12, traceId=27a2c1ccc69f405a, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:40:42.894 INFO 10 --- [http-nio-0.0.0.0-8080-exec-8,27a2c1ccc69f405a,420eeb5c026d] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.12, traceId=27a2c1ccc69f405a, duration=15 ms] 2023-11-27 13:40:43.233 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:40:43.235 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:40:44.235 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:40:44.607 INFO 10 --- [alarm-detect-3,,1b0219002f99] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_cluster_exists_inactive_server detect time range 1701063624 ~ 1701063634 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={inactive_server_count > 0=[ExpressionEvaluateResult(matchValue=0.0, metric=inactive_server_count, match=true, value=1.0, values={1701063630=1.0}, valueCount=1, labels={app=OB, ob_cluster_id=10, ob_cluster_name=darenOB, server_ips=133.197.204.3})]}, matched=true, message=null, level=2) 2023-11-27 13:40:44.911 INFO 10 --- [pool-ocp-schedules-4,,1eed8e6c4e77] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:40:45.105 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063645, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:45.105 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063645, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:45.105 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063645, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:45.105 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063645, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:45.105 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063645, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:45.457 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:40:45.479 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:40:45.515 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:40:45.830 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:40:46.235 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:40:46.236 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:40:46.525 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:40:49.016 INFO 10 --- [pool-prometheus-exporter-check2,,7d844ce1d303] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/stat, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:40:49.016 WARN 10 --- [pool-prometheus-exporter-check2,,7d844ce1d303] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/stat is inactive 2023-11-27 13:40:49.017 INFO 10 --- [pool-prometheus-exporter-check2,,7d844ce1d303] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62888/metrics/stat, message=java.net.ConnectException: Connection refused: /133.197.204.3:62888 2023-11-27 13:40:49.018 WARN 10 --- [pool-prometheus-exporter-check2,,7d844ce1d303] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62888/metrics/stat is inactive 2023-11-27 13:40:49.019 INFO 10 --- [pool-prometheus-exporter-check2,,7d844ce1d303] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/node/host, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:40:49.019 WARN 10 --- [pool-prometheus-exporter-check2,,7d844ce1d303] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/node/host is inactive 2023-11-27 13:40:49.020 INFO 10 --- [pool-prometheus-exporter-check2,,7d844ce1d303] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/obproxy, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:40:49.020 WARN 10 --- [pool-prometheus-exporter-check2,,7d844ce1d303] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/obproxy is inactive 2023-11-27 13:40:49.024 INFO 10 --- [pool-prometheus-exporter-check2,,7d844ce1d303] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/node/obproxy, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:40:49.024 WARN 10 --- [pool-prometheus-exporter-check2,,7d844ce1d303] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/node/obproxy is inactive 2023-11-27 13:40:49.025 INFO 10 --- [pool-prometheus-exporter-check2,,7d844ce1d303] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/ob/extra, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:40:49.025 WARN 10 --- [pool-prometheus-exporter-check2,,7d844ce1d303] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/ob/extra is inactive 2023-11-27 13:40:49.042 INFO 10 --- [pool-prometheus-exporter-check2,,7d844ce1d303] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/node/ob, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:40:49.042 WARN 10 --- [pool-prometheus-exporter-check2,,7d844ce1d303] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/node/ob is inactive 2023-11-27 13:40:49.049 INFO 10 --- [pool-prometheus-exporter-check2,,7d844ce1d303] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/ob/basic, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:40:49.049 WARN 10 --- [pool-prometheus-exporter-check2,,7d844ce1d303] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/ob/basic is inactive 2023-11-27 13:40:49.237 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:40:49.245 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:40:50.090 INFO 10 --- [http-nio-0.0.0.0-8080-exec-10,3f457d5f78fd40fa,efbe172931c7] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.11, traceId=3f457d5f78fd40fa, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:40:50.106 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063650, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:50.106 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063650, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:50.106 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063650, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:50.106 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063650, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:50.106 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063650, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:50.136 INFO 10 --- [http-nio-0.0.0.0-8080-exec-10,3f457d5f78fd40fa,efbe172931c7] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.11, traceId=3f457d5f78fd40fa, duration=47 ms] 2023-11-27 13:40:51.519 INFO 10 --- [pool-ocp-schedules-18,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2023-11-27 13:40:51.567 INFO 10 --- [pool-ocp-schedules-5,,e31263b4a681] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-11-27 13:40:51.583 INFO 10 --- [pool-ocp-schedules-5,,e31263b4a681] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-11-27 13:40:51.799 INFO 10 --- [pool-overview-container-6,,b98abb509e1d] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=420, alarmHistory size=15892 2023-11-27 13:40:51.879 INFO 10 --- [pool-scanner--12918-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=76 2023-11-27 13:40:52.245 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:40:52.247 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:40:52.250 INFO 10 --- [pool-ocp-schedules-12,,18188f8c8acb] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-11-27 13:40:52.250 INFO 10 --- [pool-ocp-schedules-17,,7bfaa94f2a19] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-11-27 13:40:52.252 INFO 10 --- [pool-ocp-schedules-12,,18188f8c8acb] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-11-27 13:40:52.253 INFO 10 --- [pool-ocp-schedules-17,,7bfaa94f2a19] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-11-27 13:40:52.256 INFO 10 --- [pool-ocp-schedules-19,,51fd9e937687] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-11-27 13:40:52.258 INFO 10 --- [pool-ocp-schedules-19,,51fd9e937687] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-11-27 13:40:52.301 INFO 10 --- [ForkJoinPool-47-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:52.302 INFO 10 --- [ForkJoinPool-47-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:52.302 INFO 10 --- [ForkJoinPool-47-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:52.302 INFO 10 --- [ForkJoinPool-47-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:52.304 INFO 10 --- [ForkJoinPool-47-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:52.304 INFO 10 --- [ForkJoinPool-47-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:52.305 INFO 10 --- [ForkJoinPool-47-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:52.306 INFO 10 --- [ForkJoinPool-47-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=3.1.4 2023-11-27 13:40:52.317 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:52.318 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:52.318 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:52.319 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:52.321 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:52.321 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:52.326 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, MAX(CASE `name` WHEN 'region' THEN `info` ELSE '' END ) `region`, MAX(CASE `name` WHEN 'idc' THEN `info` ELSE '' END ) `idc`, MAX(CASE `name` WHEN 'status' THEN `info` ELSE '' END ) `status`, MAX(CASE `name` WHEN 'merge_status' THEN `info` ELSE '' END ) `merge_status`, MAX(CASE `name` WHEN 'broadcast_version' THEN `value` ELSE 0 END ) `broadcast_version`, MAX(CASE `name` WHEN 'all_merged_version' THEN `value` ELSE 0 END ) `all_merged_version`, MAX(CASE `name` WHEN 'last_merged_version' THEN `value` ELSE 0 END ) `last_merged_version`, MAX(CASE `name` WHEN 'merge_start_time' THEN `value` ELSE 0 END ) `merge_start_time`, MAX(CASE `name` WHEN 'last_merged_time' THEN `value` ELSE 0 END ) `last_merged_time`, MAX(CASE `name` WHEN 'is_merge_timeout' THEN `value` ELSE 0 END ) `merge_timeout` FROM oceanbase.__all_zone WHERE `zone` <> '' GROUP BY `zone` 2023-11-27 13:40:52.333 INFO 10 --- [ForkJoinPool-47-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:52.334 INFO 10 --- [ForkJoinPool-47-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:52.334 INFO 10 --- [ForkJoinPool-47-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:52.334 INFO 10 --- [ForkJoinPool-47-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:52.338 INFO 10 --- [ForkJoinPool-47-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:52.338 INFO 10 --- [ForkJoinPool-47-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:52.339 INFO 10 --- [ForkJoinPool-47-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:52.340 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:52.341 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:52.341 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:52.342 INFO 10 --- [ForkJoinPool-47-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-11-27 13:40:52.343 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:52.346 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:52.346 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:52.347 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-11-27 13:40:52.612 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:40:52.613 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:52.613 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:52.613 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:52.614 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:52.615 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:52.615 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:52.615 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, MAX(CASE `name` WHEN 'region' THEN `info` ELSE '' END ) `region`, MAX(CASE `name` WHEN 'idc' THEN `info` ELSE '' END ) `idc`, MAX(CASE `name` WHEN 'status' THEN `info` ELSE '' END ) `status`, MAX(CASE `name` WHEN 'merge_status' THEN `info` ELSE '' END ) `merge_status`, MAX(CASE `name` WHEN 'broadcast_version' THEN `value` ELSE 0 END ) `broadcast_version`, MAX(CASE `name` WHEN 'all_merged_version' THEN `value` ELSE 0 END ) `all_merged_version`, MAX(CASE `name` WHEN 'last_merged_version' THEN `value` ELSE 0 END ) `last_merged_version`, MAX(CASE `name` WHEN 'merge_start_time' THEN `value` ELSE 0 END ) `merge_start_time`, MAX(CASE `name` WHEN 'last_merged_time' THEN `value` ELSE 0 END ) `last_merged_time`, MAX(CASE `name` WHEN 'is_merge_timeout' THEN `value` ELSE 0 END ) `merge_timeout` FROM oceanbase.__all_zone WHERE `zone` <> '' GROUP BY `zone` 2023-11-27 13:40:52.617 WARN 10 --- [ForkJoinPool-46-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-11-27 13:40:52.618 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:52.618 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:52.619 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, stop_time, start_service_time, block_migrate_in_time, id FROM oceanbase.__all_server ORDER BY zone ASC, id ASC 2023-11-27 13:40:52.620 INFO 10 --- [ForkJoinPool-46-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.206.97:2882, status=ACTIVE, startServiceTime=1701063018577428, active=true 2023-11-27 13:40:52.620 INFO 10 --- [ForkJoinPool-46-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=133.197.206.97:2882, stopTime=0, stopped=false 2023-11-27 13:40:52.620 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=IDLE, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=IDLE, startTime=null, stopTime=null} 2023-11-27 13:40:52.672 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=IDLE, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:40:52.673 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=133.197.206.97, port=2882, sqlPort=2881, version=3.1.4_10000092022071511-b4bfa011ceaef428782dcb65ae89190c40b78c2f(Jul 15 2022 11:45:14), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-27T13:30:19+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.206.97, port=2882, sqlPort=2881, version=3.1.4_10000092022071511-b4bfa011ceaef428782dcb65ae89190c40b78c2f(Jul 15 2022 11:45:14), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-27T05:30:18.577428Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:40:52.685 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=133.197.206.97, port=2882, sqlPort=2881, version=3.1.4_10000092022071511-b4bfa011ceaef428782dcb65ae89190c40b78c2f(Jul 15 2022 11:45:14), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-27T05:30:18.577428Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:40:52.718 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:40:52.720 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:52.725 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:52.725 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:52.728 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:52.733 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:52.733 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:52.735 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:52.769 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:40:52.771 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:52.772 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:52.772 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:52.772 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:52.773 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:52.774 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:52.774 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:52.775 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:52.775 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:52.776 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT cluster_id, cluster_name, created, cluster_role, cluster_status, current_scn, standby_became_primary_scn, primary_cluster_id, (time_to_usec(now(6)) - current_scn)/1000000 as delay_time, protection_mode, redo_transport_options, protection_level FROM oceanbase.v$ob_cluster 2023-11-27 13:40:52.778 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:40:52.778 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:52.779 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT switchover_status, switchover_info FROM oceanbase.v$ob_cluster 2023-11-27 13:40:52.795 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=obcluster, obClusterId=1 2023-11-27 13:40:52.795 INFO 10 --- [ForkJoinPool-46-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"133.197.206.97:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1701063583198000}, createTime=2022-11-29T14:52:53+08:00, creator=admin, obVersion=3.1.4, tenantCount=3) 2023-11-27 13:40:52.864 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:40:52.857661+08:00, lastProcessTime=2023-11-27T13:40:42.760582+08:00 2023-11-27 13:40:53.031 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : groupAggregate, now=2023-11-27T13:40:52.857661+08:00, groupByKey=0:host_unavailable:1, alarmsToSentCount=4, alarmIds=[44800, 2000006, 2000007, 2000005] 2023-11-27 13:40:53.031 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : createNotification start, groupByKey=0:host_unavailable:1 2023-11-27 13:40:53.032 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : createNotification done, groupByKey=0:host_unavailable:1, alarmIds=[44800, 2000006, 2000007, 2000005] 2023-11-27 13:40:53.032 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : groupAggregate, now=2023-11-27T13:40:52.857661+08:00, groupByKey=0:monitor_exporter_unavaliable:3, alarmsToSentCount=7, alarmIds=[44801, 44802, 44803, 44804, 44805, 44791, 44796] 2023-11-27 13:40:53.032 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : createNotification start, groupByKey=0:monitor_exporter_unavaliable:3 2023-11-27 13:40:53.033 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : createNotification done, groupByKey=0:monitor_exporter_unavaliable:3, alarmIds=[44801, 44802, 44803, 44804, 44805, 44791, 44796] 2023-11-27 13:40:53.033 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.ChannelAggregator : aggregate done, channelId=100000, now=2023-11-27T13:40:52.857661+08:00, resultsCount=2 2023-11-27 13:40:53.033 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : aggregate done, channelId=100000, alarmsCount=0, resultsCount=2 2023-11-27 13:40:53.033 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.ChannelAggregator : groupAggregate markSendSuccess, groupByKey=0:host_unavailable:1, now=2023-11-27T13:40:52.857661+08:00, alarmIds=[44800, 2000006, 2000007, 2000005] 2023-11-27 13:40:53.033 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.ChannelAggregator : groupAggregate markSendSuccess, groupByKey=0:monitor_exporter_unavaliable:3, now=2023-11-27T13:40:52.857661+08:00, alarmIds=[44801, 44802, 44803, 44804, 44805, 44791, 44796] 2023-11-27 13:40:53.033 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:40:52.857661+08:00 2023-11-27 13:40:53.143 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:40:53.145 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:53.145 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:53.145 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:53.146 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:53.152 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:53.152 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:53.153 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-11-27 13:40:53.227 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:53.227 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:53.228 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-11-27 13:40:53.230 INFO 10 --- [ForkJoinPool-46-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.204.3:2882, status=INACTIVE, startServiceTime=0, active=false 2023-11-27 13:40:53.230 INFO 10 --- [ForkJoinPool-46-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.204.10:2882, status=ACTIVE, startServiceTime=1695730092226557, active=true 2023-11-27 13:40:53.230 INFO 10 --- [ForkJoinPool-46-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=133.197.204.10:2882, stopTime=0, stopped=false 2023-11-27 13:40:53.231 INFO 10 --- [ForkJoinPool-46-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.204.11:2882, status=ACTIVE, startServiceTime=1695730305487530, active=true 2023-11-27 13:40:53.231 INFO 10 --- [ForkJoinPool-46-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=133.197.204.11:2882, stopTime=0, stopped=false 2023-11-27 13:40:53.231 INFO 10 --- [ForkJoinPool-46-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.204.12:2882, status=ACTIVE, startServiceTime=1695730736949667, active=true 2023-11-27 13:40:53.231 INFO 10 --- [ForkJoinPool-46-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=133.197.204.12:2882, stopTime=0, stopped=false 2023-11-27 13:40:53.231 INFO 10 --- [ForkJoinPool-46-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=133.197.204.13:2882, status=ACTIVE, startServiceTime=1695731005656813, active=true 2023-11-27 13:40:53.231 INFO 10 --- [ForkJoinPool-46-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=133.197.204.13:2882, stopTime=0, stopped=false 2023-11-27 13:40:53.231 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=14, name=zone6, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone6, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-27 13:40:53.271 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=14, name=zone6, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:40:53.271 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=14, ip=133.197.204.13, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=7, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T20:23:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.204.13, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=7, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:23:25.656813Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:40:53.290 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=14, ip=133.197.204.13, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=7, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:23:25.656813Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:40:53.290 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=13, name=zone5, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone5, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-27 13:40:53.314 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=13, name=zone5, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:40:53.314 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=13, ip=133.197.204.12, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=6, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T20:18:57+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.204.12, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=6, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:18:56.949667Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:40:53.333 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=13, ip=133.197.204.12, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=6, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:18:56.949667Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:40:53.333 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=12, name=zone4, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone4, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-27 13:40:53.350 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=12, name=zone4, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:40:53.350 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=12, ip=133.197.204.11, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T20:11:45+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.204.11, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:11:45.487530Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:40:53.369 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=12, ip=133.197.204.11, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:11:45.487530Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:40:53.370 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=11, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-27 13:40:53.396 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=11, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:40:53.396 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=11, ip=133.197.204.10, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=true, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T20:08:12+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.204.10, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=true, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:08:12.226557Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:40:53.424 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=11, ip=133.197.204.10, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=true, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-09-26T12:08:12.226557Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:40:53.424 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-27 13:40:53.461 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-27 13:40:53.461 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=133.197.204.3, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=2, status=UNAVAILABLE, innerStatus=INACTIVE, startTime=1970-01-01T08:00+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=133.197.204.3, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=2, status=null, innerStatus=INACTIVE, startTime=1970-01-01T00:00Z, stopTime=1970-01-01T00:00Z} 2023-11-27 13:40:53.485 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=133.197.204.3, port=2882, sqlPort=2881, version=4.1.0.0_101000022023050809-bd50a54ac52a82c5cd3b100781d5ed252822ccd0(May 8 2023 09:32:12), withRootserver=false, hostId=2, status=UNAVAILABLE, innerStatus=INACTIVE, startTime=1970-01-01T00:00Z, stopTime=1970-01-01T00:00Z}, oldStatus=UNAVAILABLE, newStatus=UNAVAILABLE 2023-11-27 13:40:53.522 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:40:53.525 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:53.526 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:53.526 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:53.527 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:53.531 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:40:53.531 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:40:53.532 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:40:53.556 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=darenOB, obClusterId=10 2023-11-27 13:40:53.557 INFO 10 --- [ForkJoinPool-46-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=darenOB, obClusterId=10, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"darenOB","ObClusterId":10,"ObRegion":"darenOB","ObRegionId":10,"ReadonlyRsList":[],"RsList":[{"address":"133.197.204.10:2882","role":"LEADER","sql_port":2881},{"address":"133.197.204.11:2882","role":"FOLLOWER","sql_port":2881},{"address":"133.197.204.12:2882","role":"FOLLOWER","sql_port":2881},{"address":"133.197.204.13:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1701063601273000}, createTime=2022-11-29T15:05:37+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-11-27 13:40:53.582 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.120.0.1:8080 2023-11-27 13:40:53.686 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.120.0.1:8080 2023-11-27 13:40:53.936 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.120.0.1:8080 2023-11-27 13:40:54.235 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:40:54.424 INFO 10 --- [alarm-detect-3,,047ca962e891] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_cluster_exists_inactive_server detect time range 1701063634 ~ 1701063644 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={inactive_server_count > 0=[ExpressionEvaluateResult(matchValue=0.0, metric=inactive_server_count, match=true, value=1.0, values={1701063640=1.0}, valueCount=1, labels={app=OB, ob_cluster_id=10, ob_cluster_name=darenOB, server_ips=133.197.204.3})]}, matched=true, message=null, level=2) 2023-11-27 13:40:54.795 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.120.0.1:8080 2023-11-27 13:40:54.938 INFO 10 --- [pool-ocp-schedules-10,,17e281220175] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:40:55.103 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063655, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:55.103 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063655, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:55.103 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063655, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:55.103 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063655, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:55.105 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063655, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:40:55.247 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:40:55.251 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:40:55.479 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:40:55.481 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:40:55.531 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:40:55.842 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:40:56.214 INFO 10 --- [pool-ocp-schedules-18,,f8b47a85a942] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:40:56.214 INFO 10 --- [pool-ocp-schedules-18,,f8b47a85a942] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:40:56.530 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:40:56.585 INFO 10 --- [pool-ocp-schedules-7,,94cd2b223b37] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2023-11-27 13:40:56.657 INFO 10 --- [pool-ocp-schedules-7,,94cd2b223b37] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:40:56.736 INFO 10 --- [pool-ocp-schedules-7,,94cd2b223b37] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:40:56.773 INFO 10 --- [pool-ocp-schedules-7,,94cd2b223b37] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:40:56.787 INFO 10 --- [pool-scanner--13011-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=76 2023-11-27 13:40:56.790 INFO 10 --- [pool-ocp-schedules-7,,94cd2b223b37] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-11-27 13:40:56.810 INFO 10 --- [pool-ocp-schedules-7,,94cd2b223b37] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-11-27 13:40:56.832 INFO 10 --- [pool-scanner--13012-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=76 2023-11-27 13:40:56.837 INFO 10 --- [pool-ocp-schedules-7,,94cd2b223b37] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2023-11-27 13:40:56.870 INFO 10 --- [pool-scanner--13013-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=76 2023-11-27 13:40:56.879 INFO 10 --- [pool-ocp-schedules-7,,94cd2b223b37] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2023-11-27 13:40:56.902 INFO 10 --- [pool-ocp-schedules-7,,94cd2b223b37] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:40:56.938 INFO 10 --- [pool-ocp-schedules-12,,af5ed481db78] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=420, alarmHistory size=15892 2023-11-27 13:40:56.946 INFO 10 --- [pool-ocp-schedules-7,,94cd2b223b37] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, count=8 2023-11-27 13:40:56.970 INFO 10 --- [pool-ocp-schedules-12,,af5ed481db78] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=420, alarmHistory size=15892 2023-11-27 13:40:57.086 INFO 10 --- [pool-ocp-schedules-12,,af5ed481db78] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=420, alarmHistory size=15892 2023-11-27 13:40:57.119 INFO 10 --- [pool-ocp-schedules-12,,af5ed481db78] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=420, alarmHistory size=15892 2023-11-27 13:40:57.172 INFO 10 --- [pool-ocp-schedules-12,,af5ed481db78] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=420, alarmHistory size=15892 2023-11-27 13:40:57.200 INFO 10 --- [pool-ocp-schedules-12,,af5ed481db78] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=420, alarmHistory size=15892 2023-11-27 13:40:58.251 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:40:58.252 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:40:59.064 INFO 10 --- [http-nio-0.0.0.0-8080-exec-9,936679e7eb1549ed,c293dcee799b] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=133.197.206.97, traceId=936679e7eb1549ed, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-11-27 13:40:59.066 INFO 10 --- [http-nio-0.0.0.0-8080-exec-9,936679e7eb1549ed,c293dcee799b] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.206.97, traceId=936679e7eb1549ed, duration=3 ms] 2023-11-27 13:41:00.000 INFO 10 --- [pool-ocp-schedules-20,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-11-27 13:41:00.000 INFO 10 --- [pool-ocp-schedules-4,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-11-27 13:41:00.000 INFO 10 --- [pool-ocp-schedules-4,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1701063660 2023-11-27 13:41:00.000 INFO 10 --- [pool-ocp-schedules-20,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-11-27 13:41:00.038 INFO 10 --- [pool-ocp-schedules-20,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-11-27 13:41:00.103 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063660, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:00.104 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063660, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:00.104 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063660, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:00.104 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063660, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:00.104 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063660, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:00.412 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:40:01.0, nextRun=2023-11-27 13:41:00.0, currentDate=Mon Nov 27 13:41:00 CST 2023 2023-11-27 13:41:00.413 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-27 13:40:01.0, nextRunTime=2023-11-27 13:41:00.0} 2023-11-27 13:41:00.469 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-11-27 13:41:00.469 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:41:00.469 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-11-27 13:41:00.475 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:40:01.0, nextRun=2023-11-27 13:41:00.0, currentDate=Mon Nov 27 13:41:00 CST 2023 2023-11-27 13:41:00.475 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-27 13:40:01.0, nextRunTime=2023-11-27 13:41:00.0} 2023-11-27 13:41:00.511 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-11-27 13:41:00.511 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:41:00.511 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-11-27 13:41:00.521 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:40:01.0, nextRun=2023-11-27 13:41:00.0, currentDate=Mon Nov 27 13:41:00 CST 2023 2023-11-27 13:41:00.522 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-27 13:40:01.0, nextRunTime=2023-11-27 13:41:00.0} 2023-11-27 13:41:00.531 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-11-27 13:41:00.532 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:41:00.532 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-11-27 13:41:00.538 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:40:31.0, nextRun=2023-11-27 13:41:00.0, currentDate=Mon Nov 27 13:41:00 CST 2023 2023-11-27 13:41:00.539 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-11-27 13:40:31.0, nextRunTime=2023-11-27 13:41:00.0} 2023-11-27 13:41:00.542 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-11-27 13:41:00.542 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:41:00.542 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-11-27 13:41:00.562 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:40:01.0, nextRun=2023-11-27 13:41:00.0, currentDate=Mon Nov 27 13:41:00 CST 2023 2023-11-27 13:41:00.562 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=1000002, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-27 13:40:01.0, nextRunTime=2023-11-27 13:41:00.0} 2023-11-27 13:41:00.565 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-11-27 13:41:00.565 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:41:00.565 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=1000002, state=RUNNING, operation=EXECUTE} 2023-11-27 13:41:00.702 INFO 10 --- [http-nio-0.0.0.0-8080-exec-2,12051ad3aa02412e,a0d8cff3532d] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.13, traceId=12051ad3aa02412e, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:41:00.742 INFO 10 --- [http-nio-0.0.0.0-8080-exec-2,12051ad3aa02412e,a0d8cff3532d] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.13, traceId=12051ad3aa02412e, duration=40 ms] 2023-11-27 13:41:00.764 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000094, operation:EXECUTE, state: RUNNING 2023-11-27 13:41:00.769 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000094, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-11-27T13:41:00.758+08:00, endTime=null} 2023-11-27 13:41:00.783 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000095, operation:EXECUTE, state: RUNNING 2023-11-27 13:41:00.788 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000095, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-11-27T13:41:00.780+08:00, endTime=null} 2023-11-27 13:41:00.808 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,e1cbfdf506a6] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000094, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000079, task_operation=execute, latest_execution_start_time=2023-11-27T13:41:00.758+08:00, sub_task_instance_id=10000094}, listMap={}}, executor=10.120.0.1 2023-11-27 13:41:00.809 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000096, operation:EXECUTE, state: RUNNING 2023-11-27 13:41:00.816 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000096, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-11-27T13:41:00.804+08:00, endTime=null} 2023-11-27 13:41:00.818 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=6 2023-11-27 13:41:00.819 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,0bddad1353c2] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000095, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000080, task_operation=execute, latest_execution_start_time=2023-11-27T13:41:00.780+08:00, sub_task_instance_id=10000095}, listMap={}}, executor=10.120.0.1 2023-11-27 13:41:00.819 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=9 2023-11-27 13:41:00.819 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-11-27 13:41:00.820 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-11-27 13:41:00.826 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-11-27 13:41:00.832 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:41:00.832 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2023-11-27 13:41:00.832 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,e1cbfdf506a6] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000094, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:41:00.834 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:41:00.848 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000097, operation:EXECUTE, state: RUNNING 2023-11-27 13:41:00.853 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000097, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-11-27T13:41:00.841+08:00, endTime=null} 2023-11-27 13:41:00.865 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,7085a6100ef8] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000096, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000081, task_operation=execute, latest_execution_start_time=2023-11-27T13:41:00.804+08:00, sub_task_instance_id=10000096}, listMap={}}, executor=10.120.0.1 2023-11-27 13:41:00.886 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000098, operation:EXECUTE, state: RUNNING 2023-11-27 13:41:00.887 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.11:62889/api/v1/log/alarms, params:null 2023-11-27 13:41:00.892 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.88:62889/api/v1/log/alarms, params:null 2023-11-27 13:41:00.893 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-11-27 13:41:00.894 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2023-11-27 13:41:00.899 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,0bddad1353c2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-11-27 13:41:00.900 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,0bddad1353c2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster darenOB, trans stat result is empty. 2023-11-27 13:41:00.901 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.12:62889/api/v1/log/alarms, params:null 2023-11-27 13:41:00.901 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2023-11-27 13:41:00.902 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000098, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-11-27T13:41:00.880+08:00, endTime=null} 2023-11-27 13:41:00.912 INFO 10 --- [ForkJoinPool-10-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.92:62889/api/v1/log/alarms, params:null 2023-11-27 13:41:00.912 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-11-27 13:41:00.926 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:41:00.929 INFO 10 --- [pool-subtask-executor32,554481c3fd1b426b,0bddad1353c2] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000095, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:41:00.934 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:41:00.939 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,d9f49541e54d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000097, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000082, task_operation=execute, latest_execution_start_time=2023-11-27T13:41:00.841+08:00, sub_task_instance_id=10000097}, listMap={}}, executor=10.120.0.1 2023-11-27 13:41:00.939 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,d9f49541e54d] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-11-27 13:41:00.943 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,7085a6100ef8] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obcluster, sql count is 0. 2023-11-27 13:41:00.943 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,7085a6100ef8] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, result is empty. 2023-11-27 13:41:00.947 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,7085a6100ef8] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster darenOB, exclude result is empty. 2023-11-27 13:41:00.970 INFO 10 --- [pool-prometheus-minute-schedule-1,,adc82109d4e5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1701063720, interval=60 2023-11-27 13:41:00.974 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,7085a6100ef8] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000096, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:41:00.975 INFO 10 --- [pool-prometheus-minute-schedule-2,,9ae70441ab74] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1701063720, interval=60 2023-11-27 13:41:01.001 INFO 10 --- [pool-prometheus-minute-schedule-1,,adc82109d4e5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=11, collectTaskCount=11 2023-11-27 13:41:01.001 INFO 10 --- [pool-prometheus-minute-schedule-1,,adc82109d4e5] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1701063660, count=11, elapsed=1699362597341 2023-11-27 13:41:01.002 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62888/metrics/stat, collectAt=1701063660, message=java.net.ConnectException: Connection refused: /133.197.206.97:62888, rootCause=ConnectException: Connection refused 2023-11-27 13:41:01.003 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/stat, collectAt=1701063660, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:01.006 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62888/metrics/stat, collectAt=1701063660, message=java.net.ConnectException: Connection refused: /133.197.206.88:62888, rootCause=ConnectException: Connection refused 2023-11-27 13:41:01.009 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,abad26107563] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000098, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000083, task_operation=execute, latest_execution_start_time=2023-11-27T13:41:00.878+08:00, sub_task_instance_id=10000098}, listMap={}}, executor=10.120.0.1 2023-11-27 13:41:01.012 INFO 10 --- [ForkJoinPool-10-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.13:62889/api/v1/log/alarms, params:null 2023-11-27 13:41:01.016 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/extra, collectAt=1701063660, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:01.044 INFO 10 --- [pool-prometheus-minute-schedule-2,,9ae70441ab74] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2023-11-27 13:41:01.044 INFO 10 --- [pool-prometheus-minute-schedule-2,,9ae70441ab74] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1701063660, count=8, elapsed=1699362597384 2023-11-27 13:41:01.044 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/stat, collectAt=1701063660, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:01.044 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62888/metrics/stat, collectAt=1701063660, message=java.net.ConnectException: Connection refused: /133.197.206.92:62888, rootCause=ConnectException: Connection refused 2023-11-27 13:41:01.045 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/stat, collectAt=1701063660, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:01.058 INFO 10 --- [ForkJoinPool-10-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.97:62889/api/v1/log/alarms, params:null 2023-11-27 13:41:01.060 INFO 10 --- [pool-1564-thread-1,554481c3fd1b426b,79533cba77cc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:darenOB, tenant:sys 2023-11-27 13:41:01.069 INFO 10 --- [pool-1564-thread-2,554481c3fd1b426b,89a3d45ec15e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:darenOB, tenant:daren 2023-11-27 13:41:01.099 INFO 10 --- [ForkJoinPool-10-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.3:62889/api/v1/log/alarms, params:null 2023-11-27 13:41:01.141 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,abad26107563] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000098, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:41:01.157 INFO 10 --- [pool-1564-thread-2,554481c3fd1b426b,89a3d45ec15e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:01.158 INFO 10 --- [pool-1564-thread-1,554481c3fd1b426b,79533cba77cc] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:01.159 INFO 10 --- [pool-1564-thread-1,554481c3fd1b426b,79533cba77cc] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:01.159 INFO 10 --- [pool-1564-thread-1,554481c3fd1b426b,79533cba77cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:01.160 INFO 10 --- [pool-1564-thread-1,554481c3fd1b426b,79533cba77cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:41:01.162 INFO 10 --- [pool-1564-thread-2,554481c3fd1b426b,89a3d45ec15e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:01.162 INFO 10 --- [pool-1564-thread-2,554481c3fd1b426b,89a3d45ec15e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:01.163 INFO 10 --- [pool-1564-thread-2,554481c3fd1b426b,89a3d45ec15e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:41:01.163 INFO 10 --- [pool-1564-thread-1,554481c3fd1b426b,79533cba77cc] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:01.163 INFO 10 --- [pool-1564-thread-1,554481c3fd1b426b,79533cba77cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:01.164 INFO 10 --- [pool-1564-thread-1,554481c3fd1b426b,79533cba77cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-11-27 13:41:01.165 INFO 10 --- [pool-1564-thread-2,554481c3fd1b426b,89a3d45ec15e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:01.165 INFO 10 --- [pool-1564-thread-2,554481c3fd1b426b,89a3d45ec15e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:01.166 INFO 10 --- [pool-1564-thread-2,554481c3fd1b426b,89a3d45ec15e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-11-27 13:41:01.199 INFO 10 --- [pool-1564-thread-1,554481c3fd1b426b,79533cba77cc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1701021601748952808, broadcast scn:1701021601748952808, saved frozen scn:1701021601748952808 2023-11-27 13:41:01.199 INFO 10 --- [pool-1564-thread-1,554481c3fd1b426b,79533cba77cc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:darenOB, tenant:sys 2023-11-27 13:41:01.202 INFO 10 --- [pool-1564-thread-2,554481c3fd1b426b,89a3d45ec15e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1701021600396785720, broadcast scn:1701021600396785720, saved frozen scn:1701021600396785720 2023-11-27 13:41:01.203 INFO 10 --- [pool-1564-thread-2,554481c3fd1b426b,89a3d45ec15e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:darenOB, tenant:daren 2023-11-27 13:41:01.217 INFO 10 --- [http-nio-0.0.0.0-8080-exec-4,770c3d649730417d,6e2a17efc0c8] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.11, traceId=770c3d649730417d, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:41:01.228 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,d9f49541e54d] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-11-27 13:41:01.251 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,d9f49541e54d] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000097, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:41:01.252 INFO 10 --- [http-nio-0.0.0.0-8080-exec-4,770c3d649730417d,6e2a17efc0c8] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.11, traceId=770c3d649730417d, duration=36 ms] 2023-11-27 13:41:01.253 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:41:01.254 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:41:01.802 INFO 10 --- [pool-overview-container-2,,5cb997942327] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=420, alarmHistory size=15892 2023-11-27 13:41:01.843 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000079, status:SUCCESS 2023-11-27 13:41:01.845 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000079, status:SUCCESS 2023-11-27 13:41:01.864 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000080, status:SUCCESS 2023-11-27 13:41:01.865 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000080, status:SUCCESS 2023-11-27 13:41:01.888 INFO 10 --- [pool-scanner--13014-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=64 2023-11-27 13:41:01.888 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000081, status:SUCCESS 2023-11-27 13:41:01.890 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000081, status:SUCCESS 2023-11-27 13:41:01.903 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000082, status:SUCCESS 2023-11-27 13:41:01.904 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000082, status:SUCCESS 2023-11-27 13:41:01.916 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000083, status:SUCCESS 2023-11-27 13:41:01.924 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000083, status:SUCCESS 2023-11-27 13:41:02.761 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:41:02.760132+08:00, lastProcessTime=2023-11-27T13:40:52.857661+08:00 2023-11-27 13:41:02.871 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:41:02.760132+08:00 2023-11-27 13:41:04.160 INFO 10 --- [pool-ocp-schedules-17,,2f819aeeb998] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=585625224722984960 2023-11-27 13:41:04.235 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:41:04.255 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:41:04.257 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:41:04.391 INFO 10 --- [alarm-detect-14,,88b9e84b5848] c.o.o.s.a.c.task.AlarmDetectMetricTask : alarmIds [44835, 44806] will be resolved. 2023-11-27 13:41:04.544 INFO 10 --- [alarm-detect-15,,00d818659b88] c.o.o.s.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans, startId: 348000002, data is empty. 2023-11-27 13:41:04.559 INFO 10 --- [alarm-detect-10,,898e0c3b3346] c.o.o.s.a.core.detect.MetricDetector : alarmType host_unavailable detect time range 1701063594 ~ 1701063654 matched, alarm-level: 1, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={host_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=host_available, match=true, value=0.0, values={1701063600=0.0}, valueCount=1, labels={app=HOST, host=133.197.206.92, host_name=localhost.localdomain, ob_cluster_id=, ob_cluster_name=, svr_ip=133.197.206.92})]}, matched=true, message=null, level=1) 2023-11-27 13:41:04.560 INFO 10 --- [alarm-detect-10,,898e0c3b3346] c.o.o.s.a.core.detect.MetricDetector : alarmType host_unavailable detect time range 1701063594 ~ 1701063654 matched, alarm-level: 1, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={host_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=host_available, match=true, value=0.0, values={1701063600=0.0}, valueCount=1, labels={app=HOST, host=133.197.206.97, host_name=localhost.localdomain, ob_cluster_id=1, ob_cluster_name=obcluster, svr_ip=133.197.206.97})]}, matched=true, message=null, level=1) 2023-11-27 13:41:04.560 INFO 10 --- [alarm-detect-10,,898e0c3b3346] c.o.o.s.a.core.detect.MetricDetector : alarmType host_unavailable detect time range 1701063594 ~ 1701063654 matched, alarm-level: 1, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={host_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=host_available, match=true, value=0.0, values={1701063600=0.0}, valueCount=1, labels={app=HOST, host=133.197.206.88, host_name=localhost.localdomain, ob_cluster_id=, ob_cluster_name=, svr_ip=133.197.206.88})]}, matched=true, message=null, level=1) 2023-11-27 13:41:04.560 INFO 10 --- [alarm-detect-10,,898e0c3b3346] c.o.o.s.a.core.detect.MetricDetector : alarmType host_unavailable detect time range 1701063594 ~ 1701063654 matched, alarm-level: 1, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={host_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=host_available, match=true, value=0.0, values={1701063600=0.0}, valueCount=1, labels={app=HOST, host=133.197.204.3, host_name=localhost.localdomain, ob_cluster_id=10, ob_cluster_name=darenOB, svr_ip=133.197.204.3})]}, matched=true, message=null, level=1) 2023-11-27 13:41:04.570 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063594 ~ 1701063654 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063600=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/obproxy, exporter_addr=http://133.197.204.3:62889/metrics/obproxy, exporter_type=OB_PROXY, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:41:04.570 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063594 ~ 1701063654 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063600=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/ob/basic, exporter_addr=http://133.197.206.97:62889/metrics/ob/basic, exporter_type=OB_CLUSTER, host=133.197.206.97, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.206.97})]}, matched=true, message=null, level=3) 2023-11-27 13:41:04.570 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063594 ~ 1701063654 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063600=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/ob, exporter_addr=http://133.197.206.97:62889/metrics/node/ob, exporter_type=OB_CLUSTER, host=133.197.206.97, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.206.97})]}, matched=true, message=null, level=3) 2023-11-27 13:41:04.570 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063594 ~ 1701063654 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063600=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://133.197.206.92:62889/metrics/stat, exporter_type=HOST_MONITOR, host=133.197.206.92, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.206.92})]}, matched=true, message=null, level=3) 2023-11-27 13:41:04.570 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063594 ~ 1701063654 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063600=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/ob/basic, exporter_addr=http://133.197.204.3:62889/metrics/ob/basic, exporter_type=OB_CLUSTER, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:41:04.571 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063594 ~ 1701063654 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063600=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://133.197.204.3:62888/metrics/stat, exporter_type=HOST_MONITOR, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:41:04.571 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063594 ~ 1701063654 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063600=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/ob/extra, exporter_addr=http://133.197.206.97:62889/metrics/ob/extra, exporter_type=OB_CLUSTER, host=133.197.206.97, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.206.97})]}, matched=true, message=null, level=3) 2023-11-27 13:41:04.571 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063594 ~ 1701063654 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063600=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://133.197.206.97:62888/metrics/stat, exporter_type=HOST_MONITOR, host=133.197.206.97, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.206.97})]}, matched=true, message=null, level=3) 2023-11-27 13:41:04.571 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063594 ~ 1701063654 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063600=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://133.197.206.92:62888/metrics/stat, exporter_type=HOST_MONITOR, host=133.197.206.92, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.206.92})]}, matched=true, message=null, level=3) 2023-11-27 13:41:04.571 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063594 ~ 1701063654 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063600=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/host, exporter_addr=http://133.197.204.3:62889/metrics/node/host, exporter_type=HOST_MONITOR, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:41:04.571 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063594 ~ 1701063654 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063600=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://133.197.204.3:62889/metrics/stat, exporter_type=HOST_MONITOR, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:41:04.571 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063594 ~ 1701063654 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063600=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://133.197.206.88:62888/metrics/stat, exporter_type=HOST_MONITOR, host=133.197.206.88, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.206.88})]}, matched=true, message=null, level=3) 2023-11-27 13:41:04.572 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063594 ~ 1701063654 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063600=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/host, exporter_addr=http://133.197.206.92:62889/metrics/node/host, exporter_type=HOST_MONITOR, host=133.197.206.92, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.206.92})]}, matched=true, message=null, level=3) 2023-11-27 13:41:04.572 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063594 ~ 1701063654 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063600=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://133.197.206.97:62889/metrics/stat, exporter_type=HOST_MONITOR, host=133.197.206.97, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.206.97})]}, matched=true, message=null, level=3) 2023-11-27 13:41:04.572 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063594 ~ 1701063654 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063600=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/host, exporter_addr=http://133.197.206.88:62889/metrics/node/host, exporter_type=HOST_MONITOR, host=133.197.206.88, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.206.88})]}, matched=true, message=null, level=3) 2023-11-27 13:41:04.572 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063594 ~ 1701063654 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063600=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/ob, exporter_addr=http://133.197.204.3:62889/metrics/node/ob, exporter_type=OB_CLUSTER, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:41:04.572 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063594 ~ 1701063654 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063600=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/ob/extra, exporter_addr=http://133.197.204.3:62889/metrics/ob/extra, exporter_type=OB_CLUSTER, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:41:04.572 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063594 ~ 1701063654 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063600=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/host, exporter_addr=http://133.197.206.97:62889/metrics/node/host, exporter_type=HOST_MONITOR, host=133.197.206.97, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.206.97})]}, matched=true, message=null, level=3) 2023-11-27 13:41:04.572 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063594 ~ 1701063654 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063600=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/node/obproxy, exporter_addr=http://133.197.204.3:62889/metrics/node/obproxy, exporter_type=OB_PROXY, host=133.197.204.3, host_name=localhost.localdomain, scrape_interval=1, svr_ip=133.197.204.3})]}, matched=true, message=null, level=3) 2023-11-27 13:41:04.572 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1701063594 ~ 1701063654 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1701063600=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://133.197.206.88:62889/metrics/stat, exporter_type=HOST_MONITOR, host=133.197.206.88, host_name=localhost.localdomain, scrape_interval=60, svr_ip=133.197.206.88})]}, matched=true, message=null, level=3) 2023-11-27 13:41:04.573 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=monitor_exporter_unavaliable, target=alarm_template_id=0:host=133.197.206.97:exporter=/metrics/ob/basic-3, currentState=Firing 2023-11-27 13:41:04.573 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=monitor_exporter_unavaliable, target=alarm_template_id=0:host=133.197.206.97:exporter=/metrics/node/ob-3, currentState=Firing 2023-11-27 13:41:04.574 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=monitor_exporter_unavaliable, target=alarm_template_id=0:host=133.197.206.92:exporter=/metrics/node/host-3, currentState=Firing 2023-11-27 13:41:04.575 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=monitor_exporter_unavaliable, target=alarm_template_id=0:host=133.197.206.88:exporter=/metrics/node/host-3, currentState=Firing 2023-11-27 13:41:04.575 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=monitor_exporter_unavaliable, target=alarm_template_id=0:host=133.197.206.97:exporter=/metrics/node/host-3, currentState=Firing 2023-11-27 13:41:04.575 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=monitor_exporter_unavaliable, needSendingCount=5, detectAlarmsCount=5, matchedTargets=[alarm_template_id=0:host=133.197.204.3:exporter=/metrics/node/host-3, alarm_template_id=0:host=133.197.204.3:exporter=/metrics/stat-3, alarm_template_id=0:host=133.197.206.88:exporter=/metrics/stat-3, alarm_template_id=0:host=133.197.206.97:exporter=/metrics/node/ob-3, alarm_template_id=0:host=133.197.206.92:exporter=/metrics/stat-3, alarm_template_id=0:host=133.197.204.3:exporter=/metrics/ob/basic-3, alarm_template_id=0:host=133.197.206.97:exporter=/metrics/stat-3, alarm_template_id=0:host=133.197.204.3:exporter=/metrics/ob/extra-3, alarm_template_id=0:host=133.197.206.97:exporter=/metrics/node/host-3, alarm_template_id=0:host=133.197.204.3:exporter=/metrics/node/ob-3, alarm_template_id=0:host=133.197.206.88:exporter=/metrics/node/host-3, alarm_template_id=0:host=133.197.204.3:exporter=/metrics/node/obproxy-3, alarm_template_id=0:host=133.197.206.97:exporter=/metrics/ob/extra-3, alarm_template_id=0:host=133.197.204.3:exporter=/metrics/obproxy-3, alarm_template_id=0:host=133.197.206.97:exporter=/metrics/ob/basic-3, alarm_template_id=0:host=133.197.206.92:exporter=/metrics/node/host-3] 2023-11-27 13:41:04.587 INFO 10 --- [alarm-detect-14,,88b9e84b5848] c.o.o.s.a.service.OcpAlarmServiceImpl : batchUpdateStatusWithLimit done, status=Inactive, tryUpdate=2, affectRow=2, affectRowHistory=2 2023-11-27 13:41:04.596 INFO 10 --- [alarm-detect-15,,00d818659b88] c.o.o.s.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans, startId: 348000002, data is empty. 2023-11-27 13:41:04.628 INFO 10 --- [alarm-detect-1,,0c0cd96fa65f] c.o.o.s.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist, startId: 348000002, data is empty. 2023-11-27 13:41:04.634 INFO 10 --- [alarm-detect-3,,f9efb6958c54] c.o.o.s.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist, startId: 348000002, data is empty. 2023-11-27 13:41:04.646 INFO 10 --- [alarm-detect-1,,0c0cd96fa65f] c.o.o.s.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist, startId: 348000002, data is empty. 2023-11-27 13:41:04.650 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.service.OcpAlarmServiceImpl : delete Done, affectRow=1, id=43456 2023-11-27 13:41:04.660 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.service.OcpAlarmServiceImpl : create alarm event Done, affectRow=1, id=2000008 2023-11-27 13:41:04.660 INFO 10 --- [alarm-detect-12,,35e9f80c683a] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_cluster_exists_inactive_server detect time range 1701063644 ~ 1701063654 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={inactive_server_count > 0=[ExpressionEvaluateResult(matchValue=0.0, metric=inactive_server_count, match=true, value=1.0, values={1701063650=1.0}, valueCount=1, labels={app=OB, ob_cluster_id=10, ob_cluster_name=darenOB, server_ips=133.197.204.3})]}, matched=true, message=null, level=2) 2023-11-27 13:41:04.666 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.service.OcpAlarmServiceImpl : create alarm event history Done, affectRow=1, id=2000008 2023-11-27 13:41:04.672 INFO 10 --- [alarm-detect-3,,f9efb6958c54] c.o.o.s.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist, startId: 348000002, data is empty. 2023-11-27 13:41:04.690 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.service.OcpAlarmServiceImpl : delete Done, affectRow=1, id=40865 2023-11-27 13:41:04.693 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.service.OcpAlarmServiceImpl : create alarm event Done, affectRow=1, id=2000009 2023-11-27 13:41:04.695 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.service.OcpAlarmServiceImpl : create alarm event history Done, affectRow=1, id=2000009 2023-11-27 13:41:04.710 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.service.OcpAlarmServiceImpl : delete Done, affectRow=1, id=41972 2023-11-27 13:41:04.712 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.service.OcpAlarmServiceImpl : create alarm event Done, affectRow=1, id=2000010 2023-11-27 13:41:04.715 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.service.OcpAlarmServiceImpl : create alarm event history Done, affectRow=1, id=2000010 2023-11-27 13:41:04.750 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.service.OcpAlarmServiceImpl : delete Done, affectRow=1, id=38715 2023-11-27 13:41:04.754 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.service.OcpAlarmServiceImpl : create alarm event Done, affectRow=1, id=2000011 2023-11-27 13:41:04.756 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.service.OcpAlarmServiceImpl : create alarm event history Done, affectRow=1, id=2000011 2023-11-27 13:41:04.772 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.service.OcpAlarmServiceImpl : delete Done, affectRow=1, id=43477 2023-11-27 13:41:04.775 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.service.OcpAlarmServiceImpl : create alarm event Done, affectRow=1, id=2000012 2023-11-27 13:41:04.777 INFO 10 --- [alarm-detect-7,,e869bb0fd607] c.o.o.s.a.service.OcpAlarmServiceImpl : create alarm event history Done, affectRow=1, id=2000012 2023-11-27 13:41:04.959 INFO 10 --- [pool-ocp-schedules-14,,6bc3d09cdce4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:41:05.109 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063665, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:05.109 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063665, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:05.109 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063665, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:05.109 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063665, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:05.109 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063665, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:05.481 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:41:05.499 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:41:05.548 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:41:05.800 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:40:05.0, nextRun=2023-11-27 13:41:05.0, currentDate=Mon Nov 27 13:41:05 CST 2023 2023-11-27 13:41:05.800 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-11-27 13:40:05.0, nextRunTime=2023-11-27 13:41:05.0} 2023-11-27 13:41:05.836 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-11-27 13:41:05.836 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:41:05.836 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-11-27 13:41:05.848 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:41:06.002 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000099, operation:EXECUTE, state: RUNNING 2023-11-27 13:41:06.010 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000099, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-11-27T13:41:05.999+08:00, endTime=null} 2023-11-27 13:41:06.050 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,607fa7edf2c5] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000099, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000084, task_operation=execute, latest_execution_start_time=2023-11-27T13:41:05.999+08:00, sub_task_instance_id=10000099}, listMap={}}, executor=10.120.0.1 2023-11-27 13:41:06.213 INFO 10 --- [pool-Check obproxy15,554481c3fd1b426b,b3e6fec6bbb5] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-11-27 13:41:06.215 WARN 10 --- [pool-Check obproxy15,554481c3fd1b426b,b3e6fec6bbb5] c.oceanbase.ocp.odpsdk.OdpJdbcOperator : Failed to check odp connection Connection {driverClassName ='com.oceanbase.jdbc.Driver', connectionProperties ='[user=root@proxysys, url=jdbc:oceanbase://133.197.204.3:2883/?connectProxy=true]'}, error is Failed to obtain JDBC Connection; nested exception is java.sql.SQLNonTransientConnectionException: Could not connect to HostAddress{host='133.197.204.3', port=2883}. Connection refused (Connection refused) 2023-11-27 13:41:06.215 WARN 10 --- [pool-Check obproxy15,554481c3fd1b426b,b3e6fec6bbb5] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to connect obproxy server, maybe caused by odp process down, wrong password, or ocp address not in private ip list and odp parameter skip_proxy_sys_private_check is not true 2023-11-27 13:41:06.215 ERROR 10 --- [pool-Check obproxy15,554481c3fd1b426b,b3e6fec6bbb5] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.UnexpectedException occurred with code error.obproxy.connection.error, and args [133.197.204.3:2883] 2023-11-27 13:41:06.225 WARN 10 --- [pool-Check obproxy15,554481c3fd1b426b,b3e6fec6bbb5] c.oceanbase.ocp.odpsdk.OdpJdbcOperator : Failed to check odp connection Connection {driverClassName ='com.oceanbase.jdbc.Driver', connectionProperties ='[user=root@proxysys, url=jdbc:oceanbase://133.197.204.3:2883/?connectProxy=true]'}, error is Failed to obtain JDBC Connection; nested exception is java.sql.SQLNonTransientConnectionException: Could not connect to HostAddress{host='133.197.204.3', port=2883}. Connection refused (Connection refused) 2023-11-27 13:41:06.225 WARN 10 --- [pool-Check obproxy15,554481c3fd1b426b,b3e6fec6bbb5] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to connect obproxy server, maybe caused by odp process down, wrong password, or ocp address not in private ip list and odp parameter skip_proxy_sys_private_check is not true 2023-11-27 13:41:06.225 ERROR 10 --- [pool-Check obproxy15,554481c3fd1b426b,b3e6fec6bbb5] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.UnexpectedException occurred with code error.obproxy.connection.error, and args [133.197.204.3:2883] 2023-11-27 13:41:06.229 INFO 10 --- [pool-Check obproxy11,554481c3fd1b426b,d5cfc39cfba3] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-11-27 13:41:06.236 INFO 10 --- [pool-Check obproxy15,554481c3fd1b426b,b3e6fec6bbb5] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 failed, will not update last active time com.oceanbase.ocp.core.exception.UnexpectedException: [OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=OBPROXY_CONNECTION_ERROR, args=133.197.204.3:2883 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_312] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_312] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_312] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_312] at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.util.ExceptionUtils.throwException(ExceptionUtils.java:90) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.util.ExceptionUtils.unExpected(ExceptionUtils.java:77) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.service.operation.obproxy.SystemOdpOperatorFactory.createOdpOperator(SystemOdpOperatorFactory.java:105) ~[ocp-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.service.operation.obproxy.SystemOdpOperatorFactory.tryToCreateOdpOperator(SystemOdpOperatorFactory.java:86) ~[ocp-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.service.operation.obproxy.ObproxyService.lambda$checkObproxyServers$20(ObproxyService.java:517) ~[ocp-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] 2023-11-27 13:41:06.246 INFO 10 --- [pool-Check obproxy13,554481c3fd1b426b,6b229bd61194] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-11-27 13:41:06.256 INFO 10 --- [pool-Check obproxy11,554481c3fd1b426b,d5cfc39cfba3] c.o.o.s.o.obproxy.ObproxyService : check obproxy 10 success, update last active time to now 2023-11-27 13:41:06.275 INFO 10 --- [pool-Check obproxy12,554481c3fd1b426b,c42e38663457] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-11-27 13:41:06.292 INFO 10 --- [pool-Check obproxy13,554481c3fd1b426b,6b229bd61194] c.o.o.s.o.obproxy.ObproxyService : check obproxy 11 success, update last active time to now 2023-11-27 13:41:06.310 INFO 10 --- [pool-Check obproxy12,554481c3fd1b426b,c42e38663457] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-11-27 13:41:06.340 INFO 10 --- [pool-Check obproxy14,554481c3fd1b426b,54495b54e865] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-11-27 13:41:06.376 INFO 10 --- [pool-Check obproxy14,554481c3fd1b426b,54495b54e865] c.o.o.s.o.obproxy.ObproxyService : check obproxy 12 success, update last active time to now 2023-11-27 13:41:06.410 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,607fa7edf2c5] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000099, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:41:06.533 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:41:07.153 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000084, status:SUCCESS 2023-11-27 13:41:07.155 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000084, status:SUCCESS 2023-11-27 13:41:07.257 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:41:07.264 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:41:07.738 INFO 10 --- [http-nio-0.0.0.0-8080-exec-10,54da1d35fec34dc7,26cba8689cc5] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.13, traceId=54da1d35fec34dc7, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:41:07.757 INFO 10 --- [http-nio-0.0.0.0-8080-exec-10,54da1d35fec34dc7,26cba8689cc5] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.13, traceId=54da1d35fec34dc7, duration=20 ms] 2023-11-27 13:41:08.844 INFO 10 --- [pool-ocp-schedules-5,,a1fbc0d8cdf6] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=7 2023-11-27 13:41:09.035 INFO 10 --- [pool-ocp-schedules-5,,a1fbc0d8cdf6] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=7 2023-11-27 13:41:09.813 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.120.0.1:8080 2023-11-27 13:41:10.104 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063670, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:10.105 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063670, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:10.105 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063670, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:10.105 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063670, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:10.106 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063670, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:10.264 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:41:10.267 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:41:11.806 INFO 10 --- [pool-overview-container-8,,9894d626dc0d] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=420, alarmHistory size=15897 2023-11-27 13:41:11.884 INFO 10 --- [pool-scanner--13142-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=8, nodesFilledCount=64 2023-11-27 13:41:12.778 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:41:12.777390+08:00, lastProcessTime=2023-11-27T13:41:02.760132+08:00 2023-11-27 13:41:12.935 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.ChannelAggregator : aggregate, pre groupAggregator is null, reset it, channelId=100000, groupByKey=0:ob_cluster_status_check_failed:1, alarmId=44806, alarmType=ob_cluster_status_check_failed, alarmTarget=alarm_template_id=0:ob_cluster=darenOB-10 2023-11-27 13:41:12.935 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : remove due alarm is not Active, alarmId=44806, alarmType=ob_cluster_status_check_failed, alarmStatus=Inactive 2023-11-27 13:41:12.935 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.ChannelAggregator : aggregate, pre groupAggregator is null, reset it, channelId=100000, groupByKey=0:ob_cluster_sync_failed:2, alarmId=44835, alarmType=ob_cluster_sync_failed, alarmTarget=alarm_template_id=0:ob_cluster=darenOB-10 2023-11-27 13:41:12.935 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : remove due alarm is not Active, alarmId=44835, alarmType=ob_cluster_sync_failed, alarmStatus=Inactive 2023-11-27 13:41:12.936 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.ChannelAggregator : aggregate, pre groupAggregator is null, reset it, channelId=100001, groupByKey=0:ob_cluster_status_check_failed:1, alarmId=44806, alarmType=ob_cluster_status_check_failed, alarmTarget=alarm_template_id=0:ob_cluster=darenOB-10 2023-11-27 13:41:12.937 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : remove due alarm is not Active, alarmId=44806, alarmType=ob_cluster_status_check_failed, alarmStatus=Inactive 2023-11-27 13:41:12.937 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.ChannelAggregator : aggregate, pre groupAggregator is null, reset it, channelId=100001, groupByKey=0:ob_cluster_sync_failed:2, alarmId=44835, alarmType=ob_cluster_sync_failed, alarmTarget=alarm_template_id=0:ob_cluster=darenOB-10 2023-11-27 13:41:12.937 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : remove due alarm is not Active, alarmId=44835, alarmType=ob_cluster_sync_failed, alarmStatus=Inactive 2023-11-27 13:41:12.938 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : aggregate done, channelId=100000, alarmsCount=7, resultsCount=2 2023-11-27 13:41:12.938 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : aggregate done, channelId=100001, alarmsCount=7, resultsCount=2 2023-11-27 13:41:12.938 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:41:12.777390+08:00 2023-11-27 13:41:13.268 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:41:13.271 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:41:13.410 INFO 10 --- [pool-metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1701063600, nodeCount=1000, affectRows=1000 2023-11-27 13:41:13.414 INFO 10 --- [pool-metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1701063600, nodeCount=1000, validCount=1000 2023-11-27 13:41:13.443 INFO 10 --- [pool-metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1701063600, nodeCount=707, affectRows=707 2023-11-27 13:41:13.447 INFO 10 --- [pool-metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1701063600, nodeCount=707, validCount=707 2023-11-27 13:41:14.235 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:41:14.506 INFO 10 --- [alarm-detect-3,,4bb8984a18dc] c.o.o.s.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist, startId: 348000002, data is empty. 2023-11-27 13:41:14.510 INFO 10 --- [alarm-detect-3,,4bb8984a18dc] c.o.o.s.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist, startId: 348000002, data is empty. 2023-11-27 13:41:14.535 INFO 10 --- [alarm-detect-3,,e18a6addaab1] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_cluster_exists_inactive_server detect time range 1701063654 ~ 1701063664 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={inactive_server_count > 0=[ExpressionEvaluateResult(matchValue=0.0, metric=inactive_server_count, match=true, value=1.0, values={1701063660=1.0}, valueCount=1, labels={app=OB, ob_cluster_id=10, ob_cluster_name=darenOB, server_ips=133.197.204.3})]}, matched=true, message=null, level=2) 2023-11-27 13:41:14.987 INFO 10 --- [pool-ocp-schedules-13,,258e5960350d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:41:15.103 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063675, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:15.104 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063675, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:15.104 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063675, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:15.106 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063675, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:15.106 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063675, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:15.499 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:41:15.521 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:41:15.563 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:41:15.851 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:41:16.273 INFO 10 --- [pool-ocp-schedules-13,,917a8f9e0d51] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:41:16.273 INFO 10 --- [pool-ocp-schedules-13,,917a8f9e0d51] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:41:16.273 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:41:16.275 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:41:16.543 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:41:16.788 INFO 10 --- [pool-ocp-schedules-3,,2480c4fed719] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-11-27 13:41:16.788 INFO 10 --- [pool-Check host status21,,9af832218cb9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=133.197.206.97, originalStatus=OFFLINE 2023-11-27 13:41:16.788 INFO 10 --- [pool-Check host status29,,24cb8e6f4e72] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=133.197.204.3, originalStatus=OFFLINE 2023-11-27 13:41:16.788 INFO 10 --- [pool-Check host status20,,d5e496d3f826] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=133.197.206.92, originalStatus=OFFLINE 2023-11-27 13:41:16.788 INFO 10 --- [pool-Check host status22,,b4fc949fe079] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=133.197.204.10, originalStatus=ONLINE 2023-11-27 13:41:16.789 INFO 10 --- [pool-Check host status23,,23ba484274da] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=133.197.204.11, originalStatus=ONLINE 2023-11-27 13:41:16.789 INFO 10 --- [pool-Check host status28,,b696445fd85c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=6, ip=133.197.204.12, originalStatus=ONLINE 2023-11-27 13:41:16.789 INFO 10 --- [pool-Check host status27,,9bf5e8d1c548] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=133.197.204.13, originalStatus=ONLINE 2023-11-27 13:41:16.792 INFO 10 --- [pool-Check host status12,,f40cd5808ebe] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=9, ip=133.197.206.88, originalStatus=OFFLINE 2023-11-27 13:41:16.799 INFO 10 --- [pool-Check host status20,,d5e496d3f826] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:41:16.804 INFO 10 --- [pool-Check host status22,,b4fc949fe079] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-11-27 13:41:16.804 INFO 10 --- [pool-Check host status23,,23ba484274da] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-11-27 13:41:16.804 INFO 10 --- [pool-Check host status27,,9bf5e8d1c548] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2023-11-27 13:41:16.805 INFO 10 --- [pool-Check host status12,,f40cd5808ebe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:41:16.807 INFO 10 --- [pool-Check host status28,,b696445fd85c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2023-11-27 13:41:16.817 INFO 10 --- [pool-Check host status29,,24cb8e6f4e72] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:41:16.821 INFO 10 --- [pool-Check host status21,,9af832218cb9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:41:16.848 INFO 10 --- [pool-Check host status12,,f40cd5808ebe] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.88:62888/api/v1/time, params:null 2023-11-27 13:41:16.851 WARN 10 --- [pool-Check host status12,,f40cd5808ebe] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check failed. host=9, ip=133.197.206.88, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.88:62888 [/133.197.206.88] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.getHostTime(AgentExecutor.java:127) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.doCheckHostStatus(HostCheckManagerImpl.java:206) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostStatus(HostCheckManagerImpl.java:168) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.88:62888 [/133.197.206.88] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 24 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 33 common frames omitted 2023-11-27 13:41:16.856 INFO 10 --- [pool-Check host status12,,f40cd5808ebe] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [9:133.197.206.88], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:41:16.857 INFO 10 --- [pool-Check host status12,,f40cd5808ebe] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:41:16.870 INFO 10 --- [pool-Check host status12,,f40cd5808ebe] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=9 2023-11-27 13:41:16.870 INFO 10 --- [pool-Check host status12,,f40cd5808ebe] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=9 2023-11-27 13:41:16.875 INFO 10 --- [pool-Check host status12,,f40cd5808ebe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:41:16.910 INFO 10 --- [pool-Check host status20,,d5e496d3f826] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.92:62888/api/v1/time, params:null 2023-11-27 13:41:16.910 INFO 10 --- [pool-Check host status22,,b4fc949fe079] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.10:62888/api/v1/time, params:null 2023-11-27 13:41:16.912 WARN 10 --- [pool-Check host status20,,d5e496d3f826] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check failed. host=3, ip=133.197.206.92, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.92:62888 [/133.197.206.92] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.getHostTime(AgentExecutor.java:127) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.doCheckHostStatus(HostCheckManagerImpl.java:206) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostStatus(HostCheckManagerImpl.java:168) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.92:62888 [/133.197.206.92] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 24 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 33 common frames omitted 2023-11-27 13:41:16.913 INFO 10 --- [pool-Check host status22,,b4fc949fe079] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=133.197.204.10, timestamp=2023-11-27T00:41:16.915188326-05:00 2023-11-27 13:41:16.914 INFO 10 --- [pool-Check host status28,,b696445fd85c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.12:62888/api/v1/time, params:null 2023-11-27 13:41:16.914 INFO 10 --- [pool-Check host status23,,23ba484274da] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.11:62888/api/v1/time, params:null 2023-11-27 13:41:16.915 INFO 10 --- [pool-Check host status22,,b4fc949fe079] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-11-27T13:41:16.913+08:00 2023-11-27 13:41:16.916 INFO 10 --- [pool-Check host status27,,9bf5e8d1c548] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.13:62888/api/v1/time, params:null 2023-11-27 13:41:16.918 INFO 10 --- [pool-Check host status20,,d5e496d3f826] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [3:133.197.206.92], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:41:16.918 INFO 10 --- [pool-Check host status20,,d5e496d3f826] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:41:16.919 INFO 10 --- [pool-Check host status23,,23ba484274da] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=133.197.204.11, timestamp=2023-11-27T00:41:16.917766889-05:00 2023-11-27 13:41:16.920 INFO 10 --- [pool-Check host status28,,b696445fd85c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=6, ip=133.197.204.12, timestamp=2023-11-27T00:41:16.920643339-05:00 2023-11-27 13:41:16.920 INFO 10 --- [pool-Check host status27,,9bf5e8d1c548] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=7, ip=133.197.204.13, timestamp=2023-11-27T00:41:16.925380276-05:00 2023-11-27 13:41:16.921 INFO 10 --- [pool-Check host status27,,9bf5e8d1c548] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2023-11-27T13:41:16.921+08:00 2023-11-27 13:41:16.922 INFO 10 --- [pool-Check host status20,,d5e496d3f826] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=3 2023-11-27 13:41:16.922 INFO 10 --- [pool-Check host status20,,d5e496d3f826] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2023-11-27 13:41:16.923 INFO 10 --- [pool-Check host status22,,b4fc949fe079] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 11 last available time 2023-11-27T13:41:16.913+08:00 and status AVAILABLE 2023-11-27 13:41:16.924 INFO 10 --- [pool-Check host status28,,b696445fd85c] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=6, lastAvailableTime=2023-11-27T13:41:16.920+08:00 2023-11-27 13:41:16.925 INFO 10 --- [pool-Check host status20,,d5e496d3f826] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:41:16.926 INFO 10 --- [pool-Check host status23,,23ba484274da] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-11-27T13:41:16.919+08:00 2023-11-27 13:41:16.929 INFO 10 --- [pool-Check host status28,,b696445fd85c] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 13 last available time 2023-11-27T13:41:16.920+08:00 and status AVAILABLE 2023-11-27 13:41:16.930 INFO 10 --- [pool-Check host status27,,9bf5e8d1c548] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 14 last available time 2023-11-27T13:41:16.921+08:00 and status AVAILABLE 2023-11-27 13:41:16.929 INFO 10 --- [pool-Check host status23,,23ba484274da] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 12 last available time 2023-11-27T13:41:16.919+08:00 and status AVAILABLE 2023-11-27 13:41:16.942 INFO 10 --- [pool-Check host status29,,24cb8e6f4e72] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.3:62888/api/v1/time, params:null 2023-11-27 13:41:16.944 WARN 10 --- [pool-Check host status29,,24cb8e6f4e72] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check failed. host=2, ip=133.197.204.3, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.204.3:62888 [/133.197.204.3] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.getHostTime(AgentExecutor.java:127) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.doCheckHostStatus(HostCheckManagerImpl.java:206) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostStatus(HostCheckManagerImpl.java:168) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.204.3:62888 [/133.197.204.3] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 24 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 33 common frames omitted 2023-11-27 13:41:16.949 INFO 10 --- [pool-Check host status28,,b696445fd85c] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=6, lastAvailableTime=2023-11-27T13:41:16.943+08:00 2023-11-27 13:41:16.950 INFO 10 --- [pool-Check host status29,,24cb8e6f4e72] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [2:133.197.204.3], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:41:16.950 INFO 10 --- [pool-Check host status29,,24cb8e6f4e72] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:41:16.951 INFO 10 --- [pool-Check host status27,,9bf5e8d1c548] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2023-11-27T13:41:16.950+08:00 2023-11-27 13:41:16.952 INFO 10 --- [pool-Check host status22,,b4fc949fe079] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-11-27T13:41:16.951+08:00 2023-11-27 13:41:16.954 INFO 10 --- [pool-Check host status29,,24cb8e6f4e72] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=2 2023-11-27 13:41:16.954 INFO 10 --- [pool-Check host status29,,24cb8e6f4e72] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2023-11-27 13:41:16.958 INFO 10 --- [pool-Check host status29,,24cb8e6f4e72] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:41:16.959 INFO 10 --- [pool-Check host status22,,b4fc949fe079] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 11 last available time 2023-11-27T13:41:16.951+08:00 and status AVAILABLE 2023-11-27 13:41:16.961 INFO 10 --- [pool-Check host status23,,23ba484274da] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-11-27T13:41:16.960+08:00 2023-11-27 13:41:16.962 INFO 10 --- [pool-Check host status28,,b696445fd85c] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 13 last available time 2023-11-27T13:41:16.943+08:00 and status AVAILABLE 2023-11-27 13:41:16.963 INFO 10 --- [pool-Check host status21,,9af832218cb9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.97:62888/api/v1/time, params:null 2023-11-27 13:41:16.971 INFO 10 --- [pool-Check host status27,,9bf5e8d1c548] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 14 last available time 2023-11-27T13:41:16.950+08:00 and status AVAILABLE 2023-11-27 13:41:16.972 WARN 10 --- [pool-Check host status21,,9af832218cb9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check failed. host=1, ip=133.197.206.97, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.97:62888 [/133.197.206.97] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.getHostTime(AgentExecutor.java:127) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.doCheckHostStatus(HostCheckManagerImpl.java:206) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostStatus(HostCheckManagerImpl.java:168) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.97:62888 [/133.197.206.97] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 24 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 33 common frames omitted 2023-11-27 13:41:16.972 INFO 10 --- [pool-Check host status23,,23ba484274da] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 12 last available time 2023-11-27T13:41:16.960+08:00 and status AVAILABLE 2023-11-27 13:41:16.977 INFO 10 --- [pool-Check host status22,,b4fc949fe079] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-11-27 13:41:16.977 INFO 10 --- [pool-Check host status23,,23ba484274da] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-11-27 13:41:16.980 INFO 10 --- [pool-Check host status28,,b696445fd85c] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=6 2023-11-27 13:41:16.988 INFO 10 --- [pool-Check host status21,,9af832218cb9] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [1:133.197.206.97], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:41:16.988 INFO 10 --- [pool-Check host status23,,23ba484274da] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-11-27 13:41:16.988 INFO 10 --- [pool-Check host status21,,9af832218cb9] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:41:16.991 INFO 10 --- [pool-Check host status21,,9af832218cb9] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=1 2023-11-27 13:41:16.991 INFO 10 --- [pool-Check host status21,,9af832218cb9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2023-11-27 13:41:16.993 INFO 10 --- [pool-Check host status22,,b4fc949fe079] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-11-27 13:41:16.994 INFO 10 --- [pool-Check host status27,,9bf5e8d1c548] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=7 2023-11-27 13:41:16.997 INFO 10 --- [pool-Check host status21,,9af832218cb9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:41:17.020 INFO 10 --- [pool-Check host status27,,9bf5e8d1c548] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2023-11-27 13:41:17.040 INFO 10 --- [pool-Check host status28,,b696445fd85c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2023-11-27 13:41:17.041 INFO 10 --- [pool-Check host status22,,b4fc949fe079] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-11-27 13:41:17.045 INFO 10 --- [pool-Check host status23,,23ba484274da] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-11-27 13:41:17.085 INFO 10 --- [pool-Check host status23,,23ba484274da] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=34 2023-11-27 13:41:17.086 INFO 10 --- [pool-Check host status22,,b4fc949fe079] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=31 2023-11-27 13:41:17.100 INFO 10 --- [pool-Check host status28,,b696445fd85c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-11-27 13:41:17.111 INFO 10 --- [pool-Check host status22,,b4fc949fe079] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=31 2023-11-27 13:41:17.112 INFO 10 --- [pool-Check host status23,,23ba484274da] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=34 2023-11-27 13:41:17.112 INFO 10 --- [pool-Check host status27,,9bf5e8d1c548] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-11-27 13:41:17.137 INFO 10 --- [pool-Check host status22,,b4fc949fe079] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-11-27 13:41:17.147 INFO 10 --- [pool-Check host status28,,b696445fd85c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=37 2023-11-27 13:41:17.164 INFO 10 --- [pool-Check host status28,,b696445fd85c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=37 2023-11-27 13:41:17.174 INFO 10 --- [pool-Check host status23,,23ba484274da] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-11-27 13:41:17.178 INFO 10 --- [pool-Check host status27,,9bf5e8d1c548] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2023-11-27 13:41:17.179 INFO 10 --- [pool-Check host status22,,b4fc949fe079] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=32 2023-11-27 13:41:17.204 INFO 10 --- [pool-Check host status28,,b696445fd85c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-11-27 13:41:17.219 INFO 10 --- [pool-Check host status23,,23ba484274da] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=35 2023-11-27 13:41:17.222 INFO 10 --- [pool-Check host status22,,b4fc949fe079] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=32 2023-11-27 13:41:17.224 INFO 10 --- [pool-Check host status27,,9bf5e8d1c548] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2023-11-27 13:41:17.231 INFO 10 --- [pool-Check host status23,,23ba484274da] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=35 2023-11-27 13:41:17.250 INFO 10 --- [pool-Check host status28,,b696445fd85c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=38 2023-11-27 13:41:17.263 INFO 10 --- [pool-Check host status23,,23ba484274da] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-11-27 13:41:17.265 INFO 10 --- [pool-Check host status22,,b4fc949fe079] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-11-27 13:41:17.274 INFO 10 --- [pool-Check host status27,,9bf5e8d1c548] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-11-27 13:41:17.278 INFO 10 --- [pool-Check host status28,,b696445fd85c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=38 2023-11-27 13:41:17.298 INFO 10 --- [pool-Check host status28,,b696445fd85c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-11-27 13:41:17.305 INFO 10 --- [pool-Check host status23,,23ba484274da] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=36 2023-11-27 13:41:17.306 INFO 10 --- [pool-Check host status22,,b4fc949fe079] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=33 2023-11-27 13:41:17.319 INFO 10 --- [pool-Check host status23,,23ba484274da] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=36 2023-11-27 13:41:17.320 INFO 10 --- [pool-Check host status22,,b4fc949fe079] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=33 2023-11-27 13:41:17.338 INFO 10 --- [pool-Check host status28,,b696445fd85c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=39 2023-11-27 13:41:17.346 INFO 10 --- [pool-Check host status27,,9bf5e8d1c548] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2023-11-27 13:41:17.385 INFO 10 --- [pool-Check host status28,,b696445fd85c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=39 2023-11-27 13:41:17.387 INFO 10 --- [pool-Check host status27,,9bf5e8d1c548] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2023-11-27 13:41:17.414 INFO 10 --- [pool-Check host status27,,9bf5e8d1c548] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://133.197.204.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-11-27 13:41:17.483 INFO 10 --- [pool-Check host status27,,9bf5e8d1c548] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2023-11-27 13:41:17.502 INFO 10 --- [pool-Check host status27,,9bf5e8d1c548] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2023-11-27 13:41:17.533 INFO 10 --- [pool-Check host status1,,272d32688236] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=133.197.206.97, originalStatus=OFFLINE 2023-11-27 13:41:17.533 INFO 10 --- [pool-Check host status25,,11d22803e824] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=133.197.204.3, originalStatus=OFFLINE 2023-11-27 13:41:17.533 INFO 10 --- [pool-Check host status3,,564f08dcd22a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=133.197.206.92, originalStatus=OFFLINE 2023-11-27 13:41:17.533 INFO 10 --- [pool-Check host status30,,5d0447e1e556] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=133.197.204.10, originalStatus=ONLINE 2023-11-27 13:41:17.533 INFO 10 --- [pool-Check host status26,,1a1f45106be4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=133.197.204.11, originalStatus=ONLINE 2023-11-27 13:41:17.533 INFO 10 --- [pool-Check host status2,,2d5dca016aed] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=6, ip=133.197.204.12, originalStatus=ONLINE 2023-11-27 13:41:17.534 INFO 10 --- [pool-Check host status17,,4c26d945f3ae] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=133.197.204.13, originalStatus=ONLINE 2023-11-27 13:41:17.534 INFO 10 --- [pool-Check host status13,,497c2526efd2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=9, ip=133.197.206.88, originalStatus=OFFLINE 2023-11-27 13:41:17.544 INFO 10 --- [pool-Check host status25,,11d22803e824] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:41:17.547 INFO 10 --- [pool-Check host status2,,2d5dca016aed] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2023-11-27 13:41:17.550 INFO 10 --- [pool-Check host status30,,5d0447e1e556] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-11-27 13:41:17.552 INFO 10 --- [pool-Check host status1,,272d32688236] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:41:17.561 INFO 10 --- [pool-Check host status13,,497c2526efd2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:41:17.561 INFO 10 --- [pool-Check host status17,,4c26d945f3ae] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2023-11-27 13:41:17.565 INFO 10 --- [pool-Check host status3,,564f08dcd22a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:41:17.622 INFO 10 --- [pool-Check host status26,,1a1f45106be4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-11-27 13:41:17.629 INFO 10 --- [pool-Check host status2,,2d5dca016aed] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.12:62888/api/v1/agent/status, params:null 2023-11-27 13:41:17.636 INFO 10 --- [pool-Check host status2,,2d5dca016aed] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=133.197.204.12 2023-11-27 13:41:17.636 INFO 10 --- [pool-Check host status2,,2d5dca016aed] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=133.197.204.12 2023-11-27 13:41:17.636 INFO 10 --- [pool-Check host status2,,2d5dca016aed] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=133.197.204.12 2023-11-27 13:41:17.647 INFO 10 --- [pool-Check host status3,,564f08dcd22a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.92:62888/api/v1/agent/status, params:null 2023-11-27 13:41:17.650 WARN 10 --- [pool-Check host status3,,564f08dcd22a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent status check failed. host=3, ip=133.197.206.92, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.92:62888 [/133.197.206.92] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.agentStatus(AgentExecutor.java:622) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostAgentStatus(HostCheckManagerImpl.java:83) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.92:62888 [/133.197.206.92] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 23 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 32 common frames omitted 2023-11-27 13:41:17.656 INFO 10 --- [pool-Check host status30,,5d0447e1e556] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.10:62888/api/v1/agent/status, params:null 2023-11-27 13:41:17.658 INFO 10 --- [pool-Check host status3,,564f08dcd22a] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [3:133.197.206.92], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:41:17.658 INFO 10 --- [pool-Check host status3,,564f08dcd22a] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:41:17.662 INFO 10 --- [pool-Check host status25,,11d22803e824] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.3:62888/api/v1/agent/status, params:null 2023-11-27 13:41:17.666 WARN 10 --- [pool-Check host status25,,11d22803e824] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent status check failed. host=2, ip=133.197.204.3, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.204.3:62888 [/133.197.204.3] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.agentStatus(AgentExecutor.java:622) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostAgentStatus(HostCheckManagerImpl.java:83) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.204.3:62888 [/133.197.204.3] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 23 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 32 common frames omitted 2023-11-27 13:41:17.667 INFO 10 --- [pool-Check host status3,,564f08dcd22a] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=3 2023-11-27 13:41:17.667 INFO 10 --- [pool-Check host status3,,564f08dcd22a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2023-11-27 13:41:17.671 INFO 10 --- [pool-Check host status3,,564f08dcd22a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-11-27 13:41:17.671 INFO 10 --- [pool-Check host status30,,5d0447e1e556] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=133.197.204.10 2023-11-27 13:41:17.671 INFO 10 --- [pool-Check host status30,,5d0447e1e556] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=133.197.204.10 2023-11-27 13:41:17.672 INFO 10 --- [pool-Check host status30,,5d0447e1e556] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=133.197.204.10 2023-11-27 13:41:17.687 INFO 10 --- [pool-Check host status13,,497c2526efd2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.88:62888/api/v1/agent/status, params:null 2023-11-27 13:41:17.689 WARN 10 --- [pool-Check host status13,,497c2526efd2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent status check failed. host=9, ip=133.197.206.88, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.88:62888 [/133.197.206.88] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.agentStatus(AgentExecutor.java:622) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostAgentStatus(HostCheckManagerImpl.java:83) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.88:62888 [/133.197.206.88] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 23 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 32 common frames omitted 2023-11-27 13:41:17.695 INFO 10 --- [pool-Check host status25,,11d22803e824] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [2:133.197.204.3], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:41:17.695 INFO 10 --- [pool-Check host status25,,11d22803e824] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:41:17.704 INFO 10 --- [pool-Check host status17,,4c26d945f3ae] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.13:62888/api/v1/agent/status, params:null 2023-11-27 13:41:17.705 INFO 10 --- [pool-Check host status25,,11d22803e824] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=2 2023-11-27 13:41:17.705 INFO 10 --- [pool-Check host status25,,11d22803e824] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2023-11-27 13:41:17.706 INFO 10 --- [pool-Check host status13,,497c2526efd2] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [9:133.197.206.88], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:41:17.706 INFO 10 --- [pool-Check host status13,,497c2526efd2] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:41:17.708 INFO 10 --- [pool-Check host status17,,4c26d945f3ae] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=133.197.204.13 2023-11-27 13:41:17.708 INFO 10 --- [pool-Check host status17,,4c26d945f3ae] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=133.197.204.13 2023-11-27 13:41:17.708 INFO 10 --- [pool-Check host status17,,4c26d945f3ae] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=133.197.204.13 2023-11-27 13:41:17.712 INFO 10 --- [pool-Check host status25,,11d22803e824] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-27 13:41:17.716 INFO 10 --- [pool-Check host status1,,272d32688236] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.206.97:62888/api/v1/agent/status, params:null 2023-11-27 13:41:17.718 WARN 10 --- [pool-Check host status1,,272d32688236] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent status check failed. host=1, ip=133.197.206.97, exception={} javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.97:62888 [/133.197.206.97] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) ~[jersey-apache-connector-2.37.jar!/:na] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) ~[jersey-client-2.37.jar!/:na] at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) ~[jersey-client-2.37.jar!/:na] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:404) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:197) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:74) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.executor.executor.AgentExecutor.agentStatus(AgentExecutor.java:622) ~[command-executor-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostAgentStatus(HostCheckManagerImpl.java:83) ~[ocp-compute-host-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$consumeIfNotRunning$0(DistributedExclusiveExecutor.java:70) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:29) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 133.197.206.97:62888 [/133.197.206.97] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar!/:4.5.13] at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ~[jersey-apache-connector-2.37.jar!/:na] ... 23 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_312] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_312] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_312] at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_312] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar!/:4.5.13] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar!/:4.5.13] ... 32 common frames omitted 2023-11-27 13:41:17.719 INFO 10 --- [pool-Check host status13,,497c2526efd2] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=9 2023-11-27 13:41:17.719 INFO 10 --- [pool-Check host status13,,497c2526efd2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=9 2023-11-27 13:41:17.721 INFO 10 --- [pool-Check host status13,,497c2526efd2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 9 2023-11-27 13:41:17.721 INFO 10 --- [pool-Check host status26,,1a1f45106be4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://133.197.204.11:62888/api/v1/agent/status, params:null 2023-11-27 13:41:17.726 INFO 10 --- [pool-Check host status1,,272d32688236] c.o.o.c.host.service.HostServiceImpl : Updated host status: host [1:133.197.206.97], from status [OFFLINE] to [OFFLINE] 2023-11-27 13:41:17.727 INFO 10 --- [pool-Check host status1,,272d32688236] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=OFFLINE) 2023-11-27 13:41:17.726 INFO 10 --- [pool-Check host status26,,1a1f45106be4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=133.197.204.11 2023-11-27 13:41:17.727 INFO 10 --- [pool-Check host status26,,1a1f45106be4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=133.197.204.11 2023-11-27 13:41:17.727 INFO 10 --- [pool-Check host status26,,1a1f45106be4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=133.197.204.11 2023-11-27 13:41:17.729 INFO 10 --- [pool-Check host status1,,272d32688236] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=1 2023-11-27 13:41:17.729 INFO 10 --- [pool-Check host status1,,272d32688236] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2023-11-27 13:41:17.733 INFO 10 --- [pool-Check host status1,,272d32688236] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-11-27 13:41:18.696 INFO 10 --- [http-nio-0.0.0.0-8080-exec-6,6c00f0caae6d4339,53c30ede9222] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.10, traceId=6c00f0caae6d4339, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:41:18.720 INFO 10 --- [http-nio-0.0.0.0-8080-exec-6,6c00f0caae6d4339,53c30ede9222] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.10, traceId=6c00f0caae6d4339, duration=24 ms] 2023-11-27 13:41:19.275 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:41:19.277 INFO 10 --- [pool-10-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:41:20.103 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063680, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:20.103 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063680, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:20.103 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063680, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:20.104 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063680, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:20.104 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063680, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:20.858 INFO 10 --- [http-nio-0.0.0.0-8080-exec-1,66cab927ceb64f2a,758639333e6c] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.10, traceId=66cab927ceb64f2a, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:41:20.861 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:40:21.0, nextRun=2023-11-27 13:41:20.0, currentDate=Mon Nov 27 13:41:20 CST 2023 2023-11-27 13:41:20.862 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-11-27 13:40:21.0, nextRunTime=2023-11-27 13:41:20.0} 2023-11-27 13:41:20.888 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-11-27 13:41:20.888 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:41:20.888 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-11-27 13:41:20.896 INFO 10 --- [http-nio-0.0.0.0-8080-exec-1,66cab927ceb64f2a,758639333e6c] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.10, traceId=66cab927ceb64f2a, duration=39 ms] 2023-11-27 13:41:21.252 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000100, operation:EXECUTE, state: RUNNING 2023-11-27 13:41:21.255 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000100, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-11-27T13:41:21.250+08:00, endTime=null} 2023-11-27 13:41:21.303 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,998ae9477542] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000100, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000085, task_operation=execute, latest_execution_start_time=2023-11-27T13:41:21.249+08:00, sub_task_instance_id=10000100}, listMap={}}, executor=10.120.0.1 2023-11-27 13:41:21.305 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,998ae9477542] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-11-27 13:41:21.306 INFO 10 --- [ForkJoinPool-50-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-11-27 13:41:21.371 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,998ae9477542] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000100, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:41:21.442 INFO 10 --- [ForkJoinPool-50-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:21.443 INFO 10 --- [ForkJoinPool-50-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:21.443 INFO 10 --- [ForkJoinPool-50-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:21.443 INFO 10 --- [ForkJoinPool-50-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:41:21.445 INFO 10 --- [ForkJoinPool-50-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:21.445 INFO 10 --- [ForkJoinPool-50-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:21.445 INFO 10 --- [ForkJoinPool-50-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT MAX(CASE `name` WHEN 'cluster' THEN `info` ELSE '' END ) `cluster`, MAX(CASE `name` WHEN 'merge_status' THEN `info` ELSE '' END ) `merge_status`, MAX(CASE `name` WHEN 'global_broadcast_version' THEN `value` ELSE 0 END ) `global_broadcast_version`, MAX(CASE `name` WHEN 'frozen_version' THEN `value` ELSE 0 END ) `frozen_version`, MAX(CASE `name` WHEN 'last_merged_version' THEN `value` ELSE 0 END ) `last_merged_version`, MAX(CASE `name` WHEN 'frozen_time' THEN `value` ELSE 0 END ) `frozen_time`, MAX(CASE `name` WHEN 'is_merge_error' THEN `value` ELSE 0 END ) `merge_error` FROM oceanbase.__all_zone WHERE `zone` = '' 2023-11-27 13:41:21.448 INFO 10 --- [ForkJoinPool-50-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:21.448 INFO 10 --- [ForkJoinPool-50-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:21.448 INFO 10 --- [ForkJoinPool-50-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select column_value from oceanbase.__all_core_table where table_name ='__all_global_stat' and column_name = 'frozen_version' 2023-11-27 13:41:21.451 INFO 10 --- [ForkJoinPool-50-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:21.451 INFO 10 --- [ForkJoinPool-50-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:21.451 INFO 10 --- [ForkJoinPool-50-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, MAX(CASE `name` WHEN 'region' THEN `info` ELSE '' END ) `region`, MAX(CASE `name` WHEN 'idc' THEN `info` ELSE '' END ) `idc`, MAX(CASE `name` WHEN 'status' THEN `info` ELSE '' END ) `status`, MAX(CASE `name` WHEN 'merge_status' THEN `info` ELSE '' END ) `merge_status`, MAX(CASE `name` WHEN 'broadcast_version' THEN `value` ELSE 0 END ) `broadcast_version`, MAX(CASE `name` WHEN 'all_merged_version' THEN `value` ELSE 0 END ) `all_merged_version`, MAX(CASE `name` WHEN 'last_merged_version' THEN `value` ELSE 0 END ) `last_merged_version`, MAX(CASE `name` WHEN 'merge_start_time' THEN `value` ELSE 0 END ) `merge_start_time`, MAX(CASE `name` WHEN 'last_merged_time' THEN `value` ELSE 0 END ) `last_merged_time`, MAX(CASE `name` WHEN 'is_merge_timeout' THEN `value` ELSE 0 END ) `merge_timeout` FROM oceanbase.__all_zone WHERE `zone` <> '' GROUP BY `zone` 2023-11-27 13:41:21.453 INFO 10 --- [ForkJoinPool-50-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : compaction collect success, cluster=ObClusterOverview(id=1, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"133.197.206.97:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1701063583198000}, createTime=2022-11-29T14:52:53+08:00, creator=admin, obVersion=3.1.4, tenantCount=3) 2023-11-27 13:41:21.668 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000085, status:SUCCESS 2023-11-27 13:41:21.670 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000085, status:SUCCESS 2023-11-27 13:41:21.801 INFO 10 --- [pool-overview-container-7,,13aa84ed058d] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=420, alarmHistory size=15897 2023-11-27 13:41:21.855 INFO 10 --- [pool-scanner--13238-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=16, nodesFilledCount=68 2023-11-27 13:41:22.277 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:41:22.278 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:41:22.769 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:41:22.768213+08:00, lastProcessTime=2023-11-27T13:41:12.777390+08:00 2023-11-27 13:41:22.797 INFO 10 --- [http-nio-0.0.0.0-8080-exec-3,424d59dbceeb459d,20615ecfe5bd] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.11, traceId=424d59dbceeb459d, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:41:22.825 INFO 10 --- [http-nio-0.0.0.0-8080-exec-3,424d59dbceeb459d,20615ecfe5bd] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.11, traceId=424d59dbceeb459d, duration=29 ms] 2023-11-27 13:41:22.870 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:41:22.768213+08:00 2023-11-27 13:41:23.618 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.120.0.1:8080 2023-11-27 13:41:23.688 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.120.0.1:8080 2023-11-27 13:41:23.945 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.120.0.1:8080 2023-11-27 13:41:24.236 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:41:24.464 INFO 10 --- [alarm-detect-8,,02f9df79a32f] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_cluster_exists_inactive_server detect time range 1701063664 ~ 1701063674 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={inactive_server_count > 0=[ExpressionEvaluateResult(matchValue=0.0, metric=inactive_server_count, match=true, value=1.0, values={1701063670=1.0}, valueCount=1, labels={app=OB, ob_cluster_id=10, ob_cluster_name=darenOB, server_ips=133.197.204.3})]}, matched=true, message=null, level=2) 2023-11-27 13:41:24.834 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.120.0.1:8080 2023-11-27 13:41:25.010 INFO 10 --- [pool-ocp-schedules-17,,3f1ce9c23148] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:41:25.103 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063685, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:25.103 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063685, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:25.103 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063685, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:25.103 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063685, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:25.103 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063685, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:25.163 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:40:25.0, nextRun=2023-11-27 13:41:25.0, currentDate=Mon Nov 27 13:41:25 CST 2023 2023-11-27 13:41:25.163 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-11-27 13:40:25.0, nextRunTime=2023-11-27 13:41:25.0} 2023-11-27 13:41:25.198 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-11-27 13:41:25.198 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:41:25.198 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-11-27 13:41:25.278 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:41:25.279 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:41:25.305 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000101, operation:EXECUTE, state: RUNNING 2023-11-27 13:41:25.307 INFO 10 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000101, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-11-27T13:41:25.303+08:00, endTime=null} 2023-11-27 13:41:25.354 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f7c8c6f2c04c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000101, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000086, task_operation=execute, latest_execution_start_time=2023-11-27T13:41:25.303+08:00, sub_task_instance_id=10000101}, listMap={host_ids=[1, 2, 3, 4, 5, 6, 7, 9]}}, executor=10.120.0.1 2023-11-27 13:41:25.403 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f7c8c6f2c04c] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=271110, maxInactiveSeconds=180 2023-11-27 13:41:25.403 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f7c8c6f2c04c] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=271110, maxInactiveSeconds=180 2023-11-27 13:41:25.403 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f7c8c6f2c04c] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=271110, maxInactiveSeconds=180 2023-11-27 13:41:25.403 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f7c8c6f2c04c] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=271110, maxInactiveSeconds=180 2023-11-27 13:41:25.403 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f7c8c6f2c04c] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=271110, maxInactiveSeconds=180 2023-11-27 13:41:25.403 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f7c8c6f2c04c] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=271110, maxInactiveSeconds=180 2023-11-27 13:41:25.403 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f7c8c6f2c04c] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=271110, maxInactiveSeconds=180 2023-11-27 13:41:25.403 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f7c8c6f2c04c] c.o.o.s.t.b.host.CheckExporterTask : Exporter address is inactive, updateTime=2023-11-24T10:22:55, inactiveSeconds=271110, maxInactiveSeconds=180 2023-11-27 13:41:25.403 WARN 10 --- [pool-subtask-executor25,554481c3fd1b426b,f7c8c6f2c04c] c.o.o.s.t.b.host.CheckExporterTask : Host [id=1, ip=133.197.206.97, status=OFFLINE] not online, skip check exporter 2023-11-27 13:41:25.403 WARN 10 --- [pool-subtask-executor25,554481c3fd1b426b,f7c8c6f2c04c] c.o.o.s.t.b.host.CheckExporterTask : Host [id=2, ip=133.197.204.3, status=OFFLINE] not online, skip check exporter 2023-11-27 13:41:25.403 WARN 10 --- [pool-subtask-executor25,554481c3fd1b426b,f7c8c6f2c04c] c.o.o.s.t.b.host.CheckExporterTask : Host [id=3, ip=133.197.206.92, status=OFFLINE] not online, skip check exporter 2023-11-27 13:41:25.404 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f7c8c6f2c04c] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=133.197.204.10] desired exporter count: 8, actual exporter count: 8 2023-11-27 13:41:25.405 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f7c8c6f2c04c] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=133.197.204.11] desired exporter count: 8, actual exporter count: 8 2023-11-27 13:41:25.406 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f7c8c6f2c04c] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=6, ip=133.197.204.12] desired exporter count: 8, actual exporter count: 8 2023-11-27 13:41:25.406 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f7c8c6f2c04c] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=7, ip=133.197.204.13] desired exporter count: 8, actual exporter count: 8 2023-11-27 13:41:25.406 WARN 10 --- [pool-subtask-executor25,554481c3fd1b426b,f7c8c6f2c04c] c.o.o.s.t.b.host.CheckExporterTask : Host [id=9, ip=133.197.206.88, status=OFFLINE] not online, skip check exporter 2023-11-27 13:41:25.411 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,f7c8c6f2c04c] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000101, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:41:25.521 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:41:25.533 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:41:25.586 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:41:25.819 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000086, status:SUCCESS 2023-11-27 13:41:25.820 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000086, status:SUCCESS 2023-11-27 13:41:25.857 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:41:26.559 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:41:28.280 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:41:28.282 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:41:30.102 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063690, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:30.102 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063690, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:30.102 INFO 10 --- [pool-metric-parse-5,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063690, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:30.103 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063690, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:30.103 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063690, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:30.181 INFO 10 --- [http-nio-0.0.0.0-8080-exec-5,7be4b25e0f194f98,032283809512] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.12, traceId=7be4b25e0f194f98, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:41:30.198 INFO 10 --- [http-nio-0.0.0.0-8080-exec-5,7be4b25e0f194f98,032283809512] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.12, traceId=7be4b25e0f194f98, duration=18 ms] 2023-11-27 13:41:30.444 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:40:30.0, nextRun=2023-11-27 13:41:30.0, currentDate=Mon Nov 27 13:41:30 CST 2023 2023-11-27 13:41:30.444 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-11-27 13:40:30.0, nextRunTime=2023-11-27 13:41:30.0} 2023-11-27 13:41:30.485 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-11-27 13:41:30.485 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:41:30.485 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:41:30.485 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-11-27 13:41:30.503 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-27 13:41:01.0, nextRun=2023-11-27 13:41:30.0, currentDate=Mon Nov 27 13:41:30 CST 2023 2023-11-27 13:41:30.503 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-11-27 13:41:01.0, nextRunTime=2023-11-27 13:41:30.0} 2023-11-27 13:41:30.506 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-11-27 13:41:30.506 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-27 13:41:30.506 INFO 10 --- [pool-task-coordinator1,554481c3fd1b426b,1bffa1b17a8d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-11-27 13:41:30.849 INFO 10 --- [http-nio-0.0.0.0-8080-exec-7,57b795a937bc4ccc,c7513096feeb] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.12, traceId=57b795a937bc4ccc, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:41:30.899 INFO 10 --- [http-nio-0.0.0.0-8080-exec-7,57b795a937bc4ccc,c7513096feeb] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.12, traceId=57b795a937bc4ccc, duration=51 ms] 2023-11-27 13:41:31.282 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:41:31.284 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:41:31.432 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000104, operation:EXECUTE, state: RUNNING 2023-11-27 13:41:31.446 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000104, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-11-27T13:41:31.429+08:00, endTime=null} 2023-11-27 13:41:31.464 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000102, operation:EXECUTE, state: RUNNING 2023-11-27 13:41:31.466 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000102, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-11-27T13:41:31.461+08:00, endTime=null} 2023-11-27 13:41:31.472 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,1afaf6832068] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000104, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=8000088, task_operation=execute, latest_execution_start_time=2023-11-27T13:41:31.429+08:00, sub_task_instance_id=10000104}, listMap={}}, executor=10.120.0.1 2023-11-27 13:41:31.472 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,1afaf6832068] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-11-27 13:41:31.499 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000103, operation:EXECUTE, state: RUNNING 2023-11-27 13:41:31.504 INFO 10 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=10000103, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-11-27T13:41:31.486+08:00, endTime=null} 2023-11-27 13:41:31.510 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000102, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=8000087, task_operation=execute, latest_execution_start_time=2023-11-27T13:41:31.460+08:00, sub_task_instance_id=10000102}, listMap={cluster_ids=[1, 2]}}, executor=10.120.0.1 2023-11-27 13:41:31.519 INFO 10 --- [pool-1740-thread-1,554481c3fd1b426b,a19e50e5483d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:darenOB, tenant:sys 2023-11-27 13:41:31.520 INFO 10 --- [pool-1740-thread-2,554481c3fd1b426b,cf8141acc523] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:darenOB, tenant:daren 2023-11-27 13:41:31.567 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=10000103, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=8000087, task_operation=execute, latest_execution_start_time=2023-11-27T13:41:31.486+08:00, sub_task_instance_id=10000103}, listMap={cluster_ids=[1, 2]}}, executor=10.120.0.1 2023-11-27 13:41:31.572 INFO 10 --- [pool-1740-thread-2,554481c3fd1b426b,cf8141acc523] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:31.573 INFO 10 --- [pool-1740-thread-2,554481c3fd1b426b,cf8141acc523] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:31.573 INFO 10 --- [pool-1740-thread-2,554481c3fd1b426b,cf8141acc523] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.574 INFO 10 --- [pool-1740-thread-1,554481c3fd1b426b,a19e50e5483d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:31.574 INFO 10 --- [pool-1740-thread-2,554481c3fd1b426b,cf8141acc523] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:41:31.576 INFO 10 --- [pool-1740-thread-1,554481c3fd1b426b,a19e50e5483d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:31.576 INFO 10 --- [pool-1740-thread-1,554481c3fd1b426b,a19e50e5483d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.577 INFO 10 --- [pool-1740-thread-1,554481c3fd1b426b,a19e50e5483d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:41:31.585 INFO 10 --- [pool-1740-thread-2,554481c3fd1b426b,cf8141acc523] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:31.585 INFO 10 --- [pool-1740-thread-2,554481c3fd1b426b,cf8141acc523] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.586 INFO 10 --- [pool-1740-thread-1,554481c3fd1b426b,a19e50e5483d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:31.586 INFO 10 --- [pool-1740-thread-1,554481c3fd1b426b,a19e50e5483d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.586 INFO 10 --- [pool-1740-thread-2,554481c3fd1b426b,cf8141acc523] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-11-27 13:41:31.588 INFO 10 --- [pool-1740-thread-1,554481c3fd1b426b,a19e50e5483d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-11-27 13:41:31.607 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:31.612 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:31.613 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.613 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:41:31.616 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:31.616 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.617 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, tenant_name, (CASE WHEN tenant_id = 1 THEN 'SYS' ELSE 'USER' END) AS tenant_type, (CASE WHEN compatibility_mode = 1 THEN 'ORACLE' ELSE 'MYSQL' END) AS compatibility_mode, zone_list, primary_zone, locality, status, locked, read_only, in_recyclebin, TIMESTAMPDIFF(SECOND, gmt_create, now()) AS exist_seconds FROM oceanbase.__all_tenant 2023-11-27 13:41:31.619 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:31.619 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.619 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1] 2023-11-27 13:41:31.624 INFO 10 --- [pool-1740-thread-1,554481c3fd1b426b,a19e50e5483d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1701021601748952808, broadcast scn:1701021601748952808, saved frozen scn:1701021601748952808 2023-11-27 13:41:31.624 INFO 10 --- [pool-1740-thread-1,554481c3fd1b426b,a19e50e5483d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:darenOB, tenant:sys 2023-11-27 13:41:31.631 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:31.631 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.631 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1] 2023-11-27 13:41:31.633 INFO 10 --- [pool-1740-thread-2,554481c3fd1b426b,cf8141acc523] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1701021600396785720, broadcast scn:1701021600396785720, saved frozen scn:1701021600396785720 2023-11-27 13:41:31.633 INFO 10 --- [pool-1740-thread-2,554481c3fd1b426b,cf8141acc523] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:darenOB, tenant:daren 2023-11-27 13:41:31.643 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:31.644 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.646 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1001] 2023-11-27 13:41:31.668 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,1afaf6832068] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-11-27 13:41:31.681 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:31.681 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.682 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1001] 2023-11-27 13:41:31.689 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:31.689 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:31.689 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.690 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:41:31.695 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:31.695 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.696 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-11-27 13:41:31.699 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:31.699 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.699 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-11-27 13:41:31.700 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:31.700 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.702 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1002] 2023-11-27 13:41:31.713 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:31.714 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.714 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:31.714 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.714 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-11-27 13:41:31.716 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1002] 2023-11-27 13:41:31.726 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-11-27 13:41:31.726 INFO 10 --- [pool-subtask-executor25,554481c3fd1b426b,1afaf6832068] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000104, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:41:31.729 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:31.729 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.730 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-11-27 13:41:31.742 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:31.742 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.743 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-11-27 13:41:31.757 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-11-27 13:41:31.808 INFO 10 --- [pool-overview-container-1,,51ba71e95a15] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=420, alarmHistory size=15897 2023-11-27 13:41:31.826 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:41:31.829 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:31.830 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:31.830 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.830 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:41:31.832 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:31.832 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.832 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-11-27 13:41:31.841 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:41:31.843 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:31.844 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:31.844 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.845 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:41:31.848 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:31.848 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.849 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-11-27 13:41:31.868 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:41:31.872 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:31.872 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:31.872 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.873 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:41:31.875 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:31.875 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.876 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:41:31.878 INFO 10 --- [pool-scanner--13331-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=16, nodesFilledCount=68 2023-11-27 13:41:31.892 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:41:31.893 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:31.894 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:31.894 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.895 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:41:31.898 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:31.898 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.899 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:41:31.906 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:41:31.908 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:31.908 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:31.908 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.909 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:41:31.910 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:31.910 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.911 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1001, ALTER_TENANT_LOCALITY] 2023-11-27 13:41:31.928 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:41:31.929 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:31.929 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:31.929 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.930 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:41:31.943 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:31.943 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.944 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1001, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:41:31.964 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:41:31.965 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:31.966 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:31.966 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.967 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:41:31.970 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:31.970 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.971 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-11-27 13:41:31.979 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:41:31.980 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:31.980 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:31.980 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.981 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:41:31.982 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:31.982 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:31.983 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-11-27 13:41:32.003 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:41:32.004 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:32.005 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:32.005 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:32.005 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000088, status:SUCCESS 2023-11-27 13:41:32.006 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:41:32.007 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000088, status:SUCCESS 2023-11-27 13:41:32.012 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:32.013 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:32.013 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:41:32.014 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:32.014 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:41:32.015 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:32.015 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:32.015 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-11-27 13:41:32.016 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:41:32.017 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:32.017 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:32.018 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, job_id, job_type, job_status, progress FROM oceanbase.__all_rootservice_job WHERE tenant_id = ? AND job_type = ? ORDER BY job_id DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-27 13:41:32.018 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-11-27 13:41:32.057 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.120.0.1 2023-11-27 13:41:32.057 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.120.0.1 2023-11-27 13:41:32.092 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:41:32.093 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:32.093 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:32.093 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:32.094 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:41:32.097 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:32.097 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:32.097 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1] 2023-11-27 13:41:32.107 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:41:32.108 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:41:32.109 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:32.110 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:32.110 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:32.111 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:41:32.118 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:32.118 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:32.119 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-11-27 13:41:32.121 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-11-27 13:41:32.122 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:32.122 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:32.125 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1001] 2023-11-27 13:41:32.131 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:41:32.134 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-11-27 13:41:32.134 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:41:32.136 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:32.136 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:32.137 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-11-27 13:41:32.137 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-11-27 13:41:32.137 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:133.197.206.97, sql port:2881 2023-11-27 13:41:32.137 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:32.138 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value FROM oceanbase.__all_virtual_sys_variable WHERE tenant_id = ?, args: [1002] 2023-11-27 13:41:32.146 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:41:32.150 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-11-27 13:41:32.150 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-11-27 13:41:32.150 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=obcluster, obClusterId=1 2023-11-27 13:41:32.151 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.120.0.1, userWhitelist=% 2023-11-27 13:41:32.155 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-11-27 13:41:32.155 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-11-27 13:41:32.155 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=darenOB, obClusterId=10 2023-11-27 13:41:32.158 INFO 10 --- [pool-subtask-executor15,554481c3fd1b426b,133f1958d92c] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000102, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:41:32.162 INFO 10 --- [pool-subtask-executor31,554481c3fd1b426b,9116904c2ca6] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 10000103, operation:EXECUTE, state: SUCCESSFUL 2023-11-27 13:41:32.767 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.service.OcpAlarmServiceImpl : batchUpdateStatusWithLimit done, status=Inactive, tryUpdate=3, affectRow=3, affectRowHistory=3 2023-11-27 13:41:33.064 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8000087, status:SUCCESS 2023-11-27 13:41:33.066 INFO 10 --- [pool-task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8000087, status:SUCCESS 2023-11-27 13:41:33.094 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : resolveByTimeout, timeoutCount=3 2023-11-27 13:41:33.144 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:41:33.143703+08:00, lastProcessTime=2023-11-27T13:41:22.768213+08:00 2023-11-27 13:41:33.240 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : remove due alarm is not Active, alarmId=2000001, alarmType=no_enough_exporter, alarmStatus=Inactive 2023-11-27 13:41:33.240 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : remove due alarm is not Active, alarmId=2000002, alarmType=no_enough_exporter, alarmStatus=Inactive 2023-11-27 13:41:33.240 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : remove due alarm is not Active, alarmId=2000003, alarmType=no_enough_exporter, alarmStatus=Inactive 2023-11-27 13:41:33.241 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : remove due alarm is not Active, alarmId=2000001, alarmType=no_enough_exporter, alarmStatus=Inactive 2023-11-27 13:41:33.241 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : remove due alarm is not Active, alarmId=2000002, alarmType=no_enough_exporter, alarmStatus=Inactive 2023-11-27 13:41:33.241 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : remove due alarm is not Active, alarmId=2000003, alarmType=no_enough_exporter, alarmStatus=Inactive 2023-11-27 13:41:33.242 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : aggregate done, channelId=100000, alarmsCount=3, resultsCount=3 2023-11-27 13:41:33.242 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : aggregate done, channelId=100001, alarmsCount=3, resultsCount=3 2023-11-27 13:41:33.242 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:41:33.143703+08:00 2023-11-27 13:41:33.916 INFO 10 --- [http-nio-0.0.0.0-8080-exec-6,10923165a0c4498f,de64bec7e0ae] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=darenOBProxy&VersionOnly=true, client=133.197.204.11, traceId=10923165a0c4498f, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,darenOBProxy,true,] 2023-11-27 13:41:33.937 INFO 10 --- [http-nio-0.0.0.0-8080-exec-6,10923165a0c4498f,de64bec7e0ae] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.204.11, traceId=10923165a0c4498f, duration=21 ms] 2023-11-27 13:41:33.976 INFO 10 --- [pool-prometheus-cache-monitor1,,83ec853b0529] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":6828,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":1707,"seriesIdKeySize":87133,"secondWriteQueue":0,"minuteWriteCacheSize":32725} 2023-11-27 13:41:33.976 INFO 10 --- [pool-prometheus-cache-monitor1,,83ec853b0529] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_zone_current_timestamp=38, ob_server_resource_memory_assigned_bytes=21, tcp_retrans=30, ob_unit_num=21, go_memstats_mspan_sys_bytes=17, go_memstats_stack_sys_bytes=17, exporter_export_metrics_bytes_total=28, node_filesystem_size_bytes=87, http_server_requests_seconds_count=40, ob_compaction_frozen_time=18, ob_system_event_total_waits=34, monagent_input_sql_duration_seconds_sum=60, observer_fd_count=20, ob_plan_cache_hit_total=28, processor_process_metrics_total=22, node_filesystem_readonly=86, ob_server_resource_disk_in_use_bytes=19, go_memstats_frees_total=17, sql_audit_input_collect_data_jumped_total=12, io_await=60, host_available=13, go_memstats_mallocs_total=17, mysql_output_write_sql_failed_count=23, ob_unit_config_max_memory_bytes=45, io_util=60, ob_system_event_detail_total_timeouts=782, ob_tenant_disk_v4_data_size=18, ob_latch_sleep_total=326, ob_tenant_unit_min_memory=41, ob_memstore_total_bytes=28, process_resident_memory_bytes=17, ob_cache_size_bytes=47, ob_tenant_unit_max_memory=41, ob_tenant_assigned_cpu_total=25, input_collect_metrics_total=29, ob_tenant_server_available_disk_free=28, ob_tenant_server_avg_cpu_used_percentage=23, node_disk_io_time_weighted_seconds_total=39, node_disk_write_time_seconds_total=39, cpu_count=30, output_write_metrics_total=15, ob_cluster_sync=175, process_open_fds=17, odp_transaction_total=22, ob_system_event_detail_time_waited_us=782, process_max_fds=17, ob_server_resource_cpu_assigned_percent=20, ocp_alarm_detect_duration_ms_count=82, go_memstats_heap_inuse_bytes=17, ob_tenant_server_max_cpu_used_percentage=23, proxy_memory_used_bytes=30, go_memstats_mcache_sys_bytes=17, ob_tenant_disk_v4_log_size=18, node_memory_memfree_bytes=29, process_exists=38, go_gc_duration_seconds=22, ob_server_resource_disk_bytes=21, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=27, node_memory_cached_bytes=29, ob_table_num=24, ob_disk_free_bytes=21, node_filesystem_avail_bytes=87, ob_backup_storage_capacity_bytes=349, ob_active_session_num=27, ob_unit_config_max_iops=45, ob_plan_cache_memory_bytes=28, node_network_transmit_errs_total=65, go_memstats_last_gc_time_seconds=17, go_gc_duration_seconds_count=17, monitor_exporter_avaliable=78, ob_tenant_server_max_memory_used_percentage=23, node_disk_read_bytes_total=39, mysql_output_write_sql_milliseconds_summary_count=25, ob_compaction_is_suspended=18, ob_tenant_unit_log_disk_size=22, go_memstats_lookups_total=17, mysql_output_write_sql_count=25, log_tailer_count=17, host_agent_process_status=21, ob_server_resource_cpu=21, go_memstats_alloc_bytes=17, ob_server_num=58, node_cpu_seconds_total_native=47, ob_tenant_unit_count=39, ob_session_all_num=25, ob_tenant_resource_min_memory=25, ob_memstore_freeze_times=28, ob_system_event_time_waited=34, ob_tenant_resource_min_cpu=25, ob_cluster_status_check=49, monagent_plugin_execute_seconds_total=41, node_net_bandwidth_bps=45, monagent_oceanbase_db_pool_idle=7, go_memstats_heap_idle_bytes=17, go_memstats_gc_sys_bytes=17, ob_server_resource_memory_bytes=21, ob_tenant_sys_partition_required_size=12, ob_latch_get_total=326, go_info=18, ob_tenant_assigned_mem_assigned=25, log_tailer_processed_file_count=24, node_load15=29, ob_tenant_server_required_size=35, ob_unit_config_min_iops=45, ob_server_partition_replica_limit=7, ob_tenant_unit_max_cpu=41, ob_latch_wait_seconds_total=326, node_memory_buffers_bytes=29, go_memstats_heap_released_bytes=17, node_network_receive_bytes_total=65, ob_memstore_freeze_trigger_bytes=28, ob_waitevent_wait_seconds_total=28, ob_server_stopped_duration_seconds=20, exposer_request_latencies_count=8, ob_role_is_rootserver=20, go_memstats_next_gc_bytes=17, mysql_output_context_timeout_discard_metrics_count=17, ob_session_active_num=25, node_filesystem_files_free=109, ob_memtable_max_snapshot_duration_seconds=27, node_disk_read_time_seconds_total=39, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=22, process_virtual_memory_bytes=17, mysql_output_write_sql_retry_count=23, ob_tenant_assigned_cpu_assigned=25, ob_tenant_unit_min_cpu=41, ob_unit_config_max_cpu=45, ob_unit_config_max_session_num=18, http_request_milliseconds_summary_count=79, monagent_oceanbase_audit_timeinfo_cache_size=7, net_connectivity=10, monagent_input_sql_duration_seconds_bucket=70, ob_latch_immediate_get_total=326, node_cpu_seconds_total=37, http_request_milliseconds_summary=82, ob_latch_immediate_miss_total=326, ob_index_error_num=16, ob_tenant_server_available_disk_total=28, ob_latch_spin_get_total=326, ob_tenant_memory_hold_bytes=18, go_memstats_buck_hash_sys_bytes=17, monagent_pipeline_report_metrics_total=18, mysql_output_write_sql_milliseconds_summary_sum=25, ob_tenant_disk_total_size=25, exposer_transferred_bytes_total=8, odp_sql_request_total=67, mysql_output_write_task_size=24, ob_tenant_context_memory_count=339, node_disk_reads_completed_total=39, exporter_export_metrics_total=28, ob_sysstat=129, ob_partition_num=13, node_ntp_offset_seconds=29, ob_latch_miss_total=326, go_memstats_heap_sys_bytes=17, ocp_meta_db_health=4, ob_disk_total_bytes=21, ob_zone_stat=38, go_memstats_heap_objects=17, go_memstats_mcache_inuse_bytes=17, go_memstats_other_sys_bytes=17, go_threads=17, http_request_milliseconds_summary_sum=79, ob_memstore_active_bytes=28, http_server_requests_seconds_sum=40, go_memstats_alloc_bytes_total=17, ob_tenant_server_partition_replica_count=35, ob_tenant_assigned_mem_total=25, node_memory_memtotal_bytes=29, node_network_transmit_drop_total=65, ob_system_event_detail_total_waits=782, odp_used_connections=28, ob_tenant_task_max_duration_seconds=25, backup_storage_capacity_used_percent=14, exposer_request_latencies=11, ob_waitevent_wait_total=28, odp_sql_cost_total=66, ob_tenant500_memory_used_bytes=20, node_network_transmit_bytes_total=65, ob_compaction_is_error=18, node_network_receive_drop_total=65, ob_server_partition_replica_count=7, monagent_pipeline_execute_total=27, node_filesystem_files=109, ob_tenant_server_avg_memory_used_percentage=23, process_start_time_seconds=17, go_memstats_heap_alloc_bytes=17, ob_tenant_resource_max_cpu=25, go_gc_duration_seconds_sum=17, go_memstats_mspan_inuse_bytes=17, mysql_output_write_sql_milliseconds_summary=28, ob_server_resource_memory_assigned_percent=20, monagent_pipeline_execute_seconds_total=27, go_memstats_sys_bytes=17, node_network_receive_errs_total=65, ob_all_session_num=27, observer_ulimit_max_fd_count=20, ob_plan_cache_access_total=28, ob_tenant_server_row_count=35, ob_tenant_memory_used_bytes=18, monagent_oceanbase_plan_cache_size=7, ob_tenant_server_data_size=35, go_memstats_gc_cpu_fraction=17, monagent_oceanbase_audit_cache_size=7, ob_tenant_context_memory_used_bytes=339, node_disk_writes_completed_total=39, node_load1=29, process_cpu_seconds_total=17, ob_tenant_resource_max_memory=25, go_memstats_stack_inuse_bytes=17, ob_tenant_user_partition_required_size=12, node_load5=29, monagent_input_sql_duration_seconds_count=60, node_disk_written_bytes_total=39, monagent_oceanbase_ash_cache_size=7, ob_tenant_context_memory_hold_bytes=339, go_goroutines=17, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=21, exposer_request_latencies_sum=8, ob_unit_config_min_cpu=45, process_virtual_memory_max_bytes=17, monagent_oceanbase_plan_timeinfo_cache_size=7, monagent_plugin_execute_total=41, ob_tenant500_memory_hold_bytes=20, exposer_scrapes_total=8, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=22, ob_compaction_start_time=18, oceanbase_connectivity=21} 2023-11-27 13:41:34.235 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:41:34.284 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:41:34.289 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:41:34.560 INFO 10 --- [alarm-detect-5,,48d5dc32cd1f] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_cluster_exists_inactive_server detect time range 1701063674 ~ 1701063684 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={inactive_server_count > 0=[ExpressionEvaluateResult(matchValue=0.0, metric=inactive_server_count, match=true, value=1.0, values={1701063680=1.0}, valueCount=1, labels={app=OB, ob_cluster_id=10, ob_cluster_name=darenOB, server_ips=133.197.204.3})]}, matched=true, message=null, level=2) 2023-11-27 13:41:35.033 INFO 10 --- [pool-ocp-schedules-4,,b95ab2385f0b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:41:35.102 INFO 10 --- [pool-prometheus-second-schedule-2,,fe7dabb1ef10] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=11, collectTaskCount=11 2023-11-27 13:41:35.102 INFO 10 --- [pool-prometheus-second-schedule-1,,8ca02d8af975] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=14, collectTaskCount=14 2023-11-27 13:41:35.103 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063695, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:35.103 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063695, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:35.103 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063695, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:35.104 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063695, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:35.104 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063695, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:35.534 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:41:35.546 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:41:35.604 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:41:35.866 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:41:36.296 INFO 10 --- [pool-ocp-schedules-18,,a95692b1cc31] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:41:36.297 INFO 10 --- [pool-ocp-schedules-18,,a95692b1cc31] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[15405f18-b190-4bf2-b8fc-c8aebe3ea16c, ab790a90-287a-4cb5-88b1-6f61d37cfe86] 2023-11-27 13:41:36.563 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:41:37.289 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:41:37.290 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:41:39.018 INFO 10 --- [pool-prometheus-exporter-check2,,4e5485b183e1] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/stat, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:41:39.018 WARN 10 --- [pool-prometheus-exporter-check2,,4e5485b183e1] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/stat is inactive 2023-11-27 13:41:39.020 INFO 10 --- [pool-prometheus-exporter-check2,,4e5485b183e1] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62888/metrics/stat, message=java.net.ConnectException: Connection refused: /133.197.204.3:62888 2023-11-27 13:41:39.020 WARN 10 --- [pool-prometheus-exporter-check2,,4e5485b183e1] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62888/metrics/stat is inactive 2023-11-27 13:41:39.021 INFO 10 --- [pool-prometheus-exporter-check2,,4e5485b183e1] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/node/host, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:41:39.021 WARN 10 --- [pool-prometheus-exporter-check2,,4e5485b183e1] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/node/host is inactive 2023-11-27 13:41:39.021 INFO 10 --- [pool-prometheus-exporter-check2,,4e5485b183e1] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/obproxy, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:41:39.022 WARN 10 --- [pool-prometheus-exporter-check2,,4e5485b183e1] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/obproxy is inactive 2023-11-27 13:41:39.025 INFO 10 --- [pool-prometheus-exporter-check2,,4e5485b183e1] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/node/obproxy, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:41:39.025 WARN 10 --- [pool-prometheus-exporter-check2,,4e5485b183e1] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/node/obproxy is inactive 2023-11-27 13:41:39.026 INFO 10 --- [pool-prometheus-exporter-check2,,4e5485b183e1] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/ob/extra, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:41:39.026 WARN 10 --- [pool-prometheus-exporter-check2,,4e5485b183e1] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/ob/extra is inactive 2023-11-27 13:41:39.027 INFO 10 --- [pool-prometheus-exporter-check2,,4e5485b183e1] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/node/ob, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:41:39.027 WARN 10 --- [pool-prometheus-exporter-check2,,4e5485b183e1] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/node/ob is inactive 2023-11-27 13:41:39.028 INFO 10 --- [pool-prometheus-exporter-check2,,4e5485b183e1] c.o.o.m.service.OcpExporterServiceImpl : exporter healthCheck failed, instance=http://133.197.204.3:62889/metrics/ob/basic, message=java.net.ConnectException: Connection refused: /133.197.204.3:62889 2023-11-27 13:41:39.028 WARN 10 --- [pool-prometheus-exporter-check2,,4e5485b183e1] .o.m.s.OcpExporterStatusCheckServiceImpl : exporter http://133.197.204.3:62889/metrics/ob/basic is inactive 2023-11-27 13:41:39.546 INFO 10 --- [pool-ocp-schedules-8,,d5bb9e22d7c7] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=3 2023-11-27 13:41:39.785 INFO 10 --- [pool-ocp-schedules-8,,d5bb9e22d7c7] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=3 2023-11-27 13:41:39.837 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.120.0.1:8080 2023-11-27 13:41:40.102 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063700, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:40.103 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063700, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:40.103 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063700, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:40.105 INFO 10 --- [pool-metric-parse-8,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063700, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:40.108 INFO 10 --- [pool-metric-parse-4,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063700, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:40.290 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:41:40.292 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:41:41.421 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2023-11-27T13:35:40.052, end=2023-11-27T13:40:41.419} 2023-11-27 13:41:41.579 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1001, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.052, end=2023-11-27T13:40:41.419} 2023-11-27 13:41:41.581 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1001, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.052, end=2023-11-27T13:40:41.419} 2023-11-27 13:41:41.583 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1001, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.052, end=2023-11-27T13:40:41.419} 2023-11-27 13:41:41.589 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1001, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.052, end=2023-11-27T13:40:41.419} 2023-11-27 13:41:41.592 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.052, end=2023-11-27T13:40:41.419} 2023-11-27 13:41:41.596 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.052, end=2023-11-27T13:40:41.419} 2023-11-27 13:41:41.598 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.052, end=2023-11-27T13:40:41.419} 2023-11-27 13:41:41.600 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.052, end=2023-11-27T13:40:41.419} 2023-11-27 13:41:41.610 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1;zone3;zone4;zone5;zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.052, end=2023-11-27T13:40:41.419} 2023-11-27 13:41:41.621 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1;zone3;zone4;zone5;zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.052, end=2023-11-27T13:40:41.419} 2023-11-27 13:41:41.631 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1;zone3;zone4;zone5;zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.052, end=2023-11-27T13:40:41.419} 2023-11-27 13:41:41.645 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1;zone3;zone4;zone5;zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.052, end=2023-11-27T13:40:41.419} 2023-11-27 13:41:41.648 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.052, end=2023-11-27T13:40:41.419} 2023-11-27 13:41:41.650 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.052, end=2023-11-27T13:40:41.419} 2023-11-27 13:41:41.652 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.052, end=2023-11-27T13:40:41.419} 2023-11-27 13:41:41.658 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.052, end=2023-11-27T13:40:41.419} 2023-11-27 13:41:41.662 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=5, name=daren, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1,zone3,zone4,zone5,zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.052, end=2023-11-27T13:40:41.419} 2023-11-27 13:41:41.669 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=5, name=daren, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1,zone3,zone4,zone5,zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.052, end=2023-11-27T13:40:41.419} 2023-11-27 13:41:41.678 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=5, name=daren, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1,zone3,zone4,zone5,zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.052, end=2023-11-27T13:40:41.419} 2023-11-27 13:41:41.683 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=5, name=daren, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1,zone3,zone4,zone5,zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.052, end=2023-11-27T13:40:41.419} 2023-11-27 13:41:41.683 INFO 10 --- [pool-14-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2023-11-27 13:41:41.802 INFO 10 --- [pool-overview-container-2,,78649144e5f2] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=420, alarmHistory size=15897 2023-11-27 13:41:41.872 INFO 10 --- [pool-scanner--13424-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=16, nodesFilledCount=68 2023-11-27 13:41:42.726 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.service.OcpAlarmServiceImpl : batchUpdateStatusWithLimit done, status=Inactive, tryUpdate=1, affectRow=1, affectRowHistory=1 2023-11-27 13:41:42.936 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : resolveByTimeout, timeoutCount=1 2023-11-27 13:41:42.985 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-27T13:41:42.984957+08:00, lastProcessTime=2023-11-27T13:41:33.143703+08:00 2023-11-27 13:41:43.097 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : remove due alarm is not Active, alarmId=2000004, alarmType=no_enough_exporter, alarmStatus=Inactive 2023-11-27 13:41:43.098 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.c.aggregate.GroupAggregator : remove due alarm is not Active, alarmId=2000004, alarmType=no_enough_exporter, alarmStatus=Inactive 2023-11-27 13:41:43.099 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : aggregate done, channelId=100000, alarmsCount=1, resultsCount=1 2023-11-27 13:41:43.099 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : aggregate done, channelId=100001, alarmsCount=1, resultsCount=1 2023-11-27 13:41:43.099 INFO 10 --- [alarm-task-2,,f3328dd1ed63] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-27T13:41:42.984957+08:00 2023-11-27 13:41:43.292 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:41:43.294 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:41:44.235 INFO 10 --- [alarm-task-1,,a525f2312e35] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-27 13:41:44.463 INFO 10 --- [alarm-detect-14,,ab8c1aa3884d] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_cluster_exists_inactive_server detect time range 1701063684 ~ 1701063694 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={inactive_server_count > 0=[ExpressionEvaluateResult(matchValue=0.0, metric=inactive_server_count, match=true, value=1.0, values={1701063690=1.0}, valueCount=1, labels={app=OB, ob_cluster_id=10, ob_cluster_name=darenOB, server_ips=133.197.204.3})]}, matched=true, message=null, level=2) 2023-11-27 13:41:44.709 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2023-11-27T13:35:40.097, end=2023-11-27T13:40:44.703} 2023-11-27 13:41:44.783 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1001, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.097, end=2023-11-27T13:40:44.703} 2023-11-27 13:41:44.793 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1001, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.097, end=2023-11-27T13:40:44.703} 2023-11-27 13:41:44.802 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1001, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.097, end=2023-11-27T13:40:44.703} 2023-11-27 13:41:44.811 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1001, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.097, end=2023-11-27T13:40:44.703} 2023-11-27 13:41:44.812 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-11-27 13:41:44.848 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=5, name=daren, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1,zone3,zone4,zone5,zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.097, end=2023-11-27T13:40:44.703} 2023-11-27 13:41:44.866 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 11 diagnose targets Pair(left=Tenant(id=5, name=daren, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1,zone3,zone4,zone5,zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.097, end=2023-11-27T13:40:44.703} 2023-11-27 13:41:44.874 INFO 10 --- [pool-sql_perf_diagnoser3,,] .s.d.i.d.PerformanceDegradationDiagnoser : 97C71B9069E7D1F3C25197BCD97D7130, diagParam:MINUTES, 20 2023-11-27 13:41:45.058 INFO 10 --- [pool-ocp-schedules-13,,edc68dd60903] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-27 13:41:45.087 INFO 10 --- [pool-sql_perf_diagnoser3,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:45.088 INFO 10 --- [pool-sql_perf_diagnoser3,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:45.088 INFO 10 --- [pool-sql_perf_diagnoser3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:45.089 INFO 10 --- [pool-sql_perf_diagnoser3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:41:45.097 INFO 10 --- [pool-sql_perf_diagnoser3,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:45.097 INFO 10 --- [pool-sql_perf_diagnoser3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:45.098 INFO 10 --- [pool-sql_perf_diagnoser3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT/*+ QUERY_TIMEOUT(30000000) */ con_id as tenant_id, object_id as database_id, object_name as database_name FROM oceanbase.cdb_objects where con_id = ? and OBJECT_TYPE = 'DATABASE' , args: [1002] 2023-11-27 13:41:45.103 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063705, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:45.103 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063705, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:45.103 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063705, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:45.103 INFO 10 --- [pool-metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063705, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:45.104 INFO 10 --- [pool-metric-parse-1,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063705, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:45.418 INFO 10 --- [pool-sql_perf_diagnoser3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:41:45.419 INFO 10 --- [pool-sql_perf_diagnoser3,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:45.420 INFO 10 --- [pool-sql_perf_diagnoser3,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:45.420 INFO 10 --- [pool-sql_perf_diagnoser3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:45.421 INFO 10 --- [pool-sql_perf_diagnoser3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:41:45.424 INFO 10 --- [pool-sql_perf_diagnoser3,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:45.424 INFO 10 --- [pool-sql_perf_diagnoser3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:45.425 INFO 10 --- [pool-sql_perf_diagnoser3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-11-27 13:41:45.446 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=5, name=daren, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1,zone3,zone4,zone5,zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.097, end=2023-11-27T13:40:44.703} 2023-11-27 13:41:45.462 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 13 diagnose targets Pair(left=Tenant(id=5, name=daren, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1,zone3,zone4,zone5,zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.097, end=2023-11-27T13:40:44.703} 2023-11-27 13:41:45.526 INFO 10 --- [pool-sql_perf_diagnoser4,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:45.527 INFO 10 --- [pool-sql_perf_diagnoser4,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:45.527 INFO 10 --- [pool-sql_perf_diagnoser4,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:45.528 INFO 10 --- [pool-sql_perf_diagnoser1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:45.528 INFO 10 --- [pool-sql_perf_diagnoser4,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:41:45.528 INFO 10 --- [pool-sql_perf_diagnoser1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:45.528 INFO 10 --- [pool-sql_perf_diagnoser1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:45.529 INFO 10 --- [pool-sql_perf_diagnoser1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:41:45.534 INFO 10 --- [pool-sql_perf_diagnoser4,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:45.534 INFO 10 --- [pool-sql_perf_diagnoser4,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:45.534 INFO 10 --- [pool-sql_perf_diagnoser1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:45.534 INFO 10 --- [pool-sql_perf_diagnoser1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:45.535 INFO 10 --- [pool-sql_perf_diagnoser4,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT/*+ QUERY_TIMEOUT(30000000) */ con_id as tenant_id, object_id as database_id, object_name as database_name FROM oceanbase.cdb_objects where con_id = ? and OBJECT_TYPE = 'DATABASE' , args: [1002] 2023-11-27 13:41:45.535 INFO 10 --- [pool-sql_perf_diagnoser1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT/*+ QUERY_TIMEOUT(30000000) */ con_id as tenant_id, object_id as database_id, object_name as database_name FROM oceanbase.cdb_objects where con_id = ? and OBJECT_TYPE = 'DATABASE' , args: [1002] 2023-11-27 13:41:45.546 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:41:45.557 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.120.0.1:8080 2023-11-27 13:41:45.603 INFO 10 --- [pool-sql_perf_diagnoser1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:41:45.605 INFO 10 --- [pool-sql_perf_diagnoser1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:45.605 INFO 10 --- [pool-sql_perf_diagnoser1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:45.605 INFO 10 --- [pool-sql_perf_diagnoser1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:45.606 INFO 10 --- [pool-sql_perf_diagnoser1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:41:45.610 INFO 10 --- [pool-sql_perf_diagnoser1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:45.610 INFO 10 --- [pool-sql_perf_diagnoser1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:45.611 INFO 10 --- [pool-sql_perf_diagnoser1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-11-27 13:41:45.615 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.120.0.1:8080 2023-11-27 13:41:45.707 INFO 10 --- [pool-sql_perf_diagnoser4,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=darenOB, tenantName=sys, dbUser=ocp_monitor 2023-11-27 13:41:45.708 INFO 10 --- [pool-sql_perf_diagnoser4,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:45.708 INFO 10 --- [pool-sql_perf_diagnoser4,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:45.708 INFO 10 --- [pool-sql_perf_diagnoser4,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:45.709 INFO 10 --- [pool-sql_perf_diagnoser4,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:41:45.713 INFO 10 --- [pool-sql_perf_diagnoser4,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:45.713 INFO 10 --- [pool-sql_perf_diagnoser4,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:45.714 INFO 10 --- [pool-sql_perf_diagnoser4,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-11-27 13:41:45.722 INFO 10 --- [pool-sql_perf_diagnoser7,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:45.722 INFO 10 --- [pool-sql_perf_diagnoser7,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:45.722 INFO 10 --- [pool-sql_perf_diagnoser7,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:45.723 INFO 10 --- [pool-sql_perf_diagnoser7,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-27 13:41:45.726 INFO 10 --- [pool-sql_perf_diagnoser7,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-27 13:41:45.727 INFO 10 --- [pool-sql_perf_diagnoser7,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-27 13:41:45.727 INFO 10 --- [pool-sql_perf_diagnoser7,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-11-27 13:41:45.735 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-11-27 13:41:45.751 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.097, end=2023-11-27T13:40:44.703} 2023-11-27 13:41:45.768 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.097, end=2023-11-27T13:40:44.703} 2023-11-27 13:41:45.783 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.097, end=2023-11-27T13:40:44.703} 2023-11-27 13:41:45.796 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.097, end=2023-11-27T13:40:44.703} 2023-11-27 13:41:45.797 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-11-27 13:41:45.822 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1;zone3;zone4;zone5;zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.097, end=2023-11-27T13:40:44.703} 2023-11-27 13:41:45.852 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 7 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1;zone3;zone4;zone5;zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.097, end=2023-11-27T13:40:44.703} 2023-11-27 13:41:45.874 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.120.0.1:8080 2023-11-27 13:41:45.919 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1;zone3;zone4;zone5;zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.097, end=2023-11-27T13:40:44.703} 2023-11-27 13:41:45.952 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 30 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=darenOB, obClusterId=10, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T15:05:58+08:00, primaryZone=zone1;zone3;zone4;zone5;zone6, zoneList=zone1;zone3;zone4;zone5;zone6, locality=FULL{1}@zone1, FULL{1}@zone3, FULL{1}@zone4, FULL{1}@zone5, FULL{1}@zone6, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.097, end=2023-11-27T13:40:44.703} 2023-11-27 13:41:45.984 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-11-27 13:41:45.994 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.097, end=2023-11-27T13:40:44.703} 2023-11-27 13:41:46.013 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.097, end=2023-11-27T13:40:44.703} 2023-11-27 13:41:46.024 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.097, end=2023-11-27T13:40:44.703} 2023-11-27 13:41:46.049 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=obcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2022-11-29T14:54:10+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=3.1.4, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-11-27T13:35:40.097, end=2023-11-27T13:40:44.703} 2023-11-27 13:41:46.050 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-11-27 13:41:46.050 INFO 10 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2023-11-27 13:41:46.295 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:41:46.300 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:41:46.578 INFO 10 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.120.0.1:8080 2023-11-27 13:41:48.920 INFO 10 --- [http-nio-0.0.0.0-8080-exec-1,c84923ee5e3742d0,4bc922fc6686] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=133.197.206.97, traceId=c84923ee5e3742d0, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-11-27 13:41:48.925 INFO 10 --- [http-nio-0.0.0.0-8080-exec-1,c84923ee5e3742d0,4bc922fc6686] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=133.197.206.97, traceId=c84923ee5e3742d0, duration=6 ms] 2023-11-27 13:41:49.300 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.120.0.1:8080 refreshes policies... 2023-11-27 13:41:49.304 INFO 10 --- [pool-10-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-27 13:41:49.789 INFO 10 --- [pool-ocp-schedules-5,,86d2389946e1] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-11-27 13:41:49.969 INFO 10 --- [pool-ocp-schedules-5,,86d2389946e1] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-11-27 13:41:50.103 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/ob/basic, collectAt=1701063710, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:50.103 INFO 10 --- [pool-metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/host, collectAt=1701063710, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:50.108 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.97:62889/metrics/node/ob, collectAt=1701063710, message=java.net.ConnectException: Connection refused: /133.197.206.97:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:50.108 INFO 10 --- [pool-metric-parse-2,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.92:62889/metrics/node/host, collectAt=1701063710, message=java.net.ConnectException: Connection refused: /133.197.206.92:62889, rootCause=ConnectException: Connection refused 2023-11-27 13:41:50.108 INFO 10 --- [pool-metric-parse-7,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://133.197.206.88:62889/metrics/node/host, collectAt=1701063710, message=java.net.ConnectException: Connection refused: /133.197.206.88:62889, rootCause=ConnectException: Connection refused