2023-11-30 16:12:48.266 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:12:48.268 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:12:49.503 INFO 12 --- [pool-ocp-schedules-10,,88f6caabba32] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[997e1fae-052d-4a24-a32c-2dd49f574f28, a1cfb9f7-f190-4ebe-9f7b-70b8f35ffcf1] 2023-11-30 16:12:49.504 INFO 12 --- [pool-ocp-schedules-10,,88f6caabba32] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[997e1fae-052d-4a24-a32c-2dd49f574f28, a1cfb9f7-f190-4ebe-9f7b-70b8f35ffcf1] 2023-11-30 16:12:50.664 INFO 12 --- [alarm-task-1,,89b05ccdc1eb] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-30 16:12:50.686 INFO 12 --- [alarm-task-2,,8ed9b273a5a2] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-30T16:12:50.685447+08:00, lastProcessTime=2023-11-30T16:12:40.681397+08:00 2023-11-30 16:12:50.810 INFO 12 --- [alarm-task-2,,8ed9b273a5a2] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-30T16:12:50.685447+08:00 2023-11-30 16:12:50.890 INFO 12 --- [alarm-detect-9,,3dcb0ad491cf] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1701331950 ~ 1701331960 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.617, values={1701331960=1.617}, valueCount=1, labels={host=10.38.68.74, host_name=az-rta-prod-database-database-mysql-13, ob_cluster_id=1, ob_cluster_name=local_v4, obzone=zone1, svr_ip=10.38.68.74})]}, matched=true, message=null, level=3) 2023-11-30 16:12:50.890 INFO 12 --- [alarm-detect-9,,3dcb0ad491cf] .o.o.s.a.c.d.AlarmRuleDetectStateManager : alarmType ob_host_load1_per_cpu_over_threshold targets.size 1 was marked as not match 2023-11-30 16:12:51.268 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:12:51.274 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:12:51.358 INFO 12 --- [pool-overview-container-3,,379723ba9faf] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=3, alarmHistory size=6 2023-11-30 16:12:51.639 INFO 12 --- [pool-ocp-schedules-9,,6fba47485b5d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-30 16:12:52.171 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.38.66.53:8080 2023-11-30 16:12:52.335 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.38.66.53:8080 2023-11-30 16:12:52.348 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.38.66.53:8080 2023-11-30 16:12:52.376 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.38.66.53:8080 2023-11-30 16:12:53.351 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.38.66.53:8080 2023-11-30 16:12:54.274 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:12:54.276 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:12:55.105 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.38.66.53:8080 2023-11-30 16:12:57.225 INFO 12 --- [http-nio-0.0.0.0-8080-exec-1,,] c.o.o.c.s.util.CsrfRequestMatcher : CSRF token required for API [DELETE /api/v2/software-packages/4, client 10.38.68.74, traceId ] 2023-11-30 16:12:57.228 INFO 12 --- [http-nio-0.0.0.0-8080-exec-1,f4f81cdba2d94ad1,41e4dcf0bb8a] c.o.o.server.trace.RequestTracingAspect : API: [DELETE /api/v2/software-packages/4?id=4, client=10.38.68.74, traceId=f4f81cdba2d94ad1, method=NoDataResponse com.oceanbase.ocp.server.controller.SoftwarePackageController.deletePackage(long), args=4,] 2023-11-30 16:12:57.276 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:12:57.284 INFO 12 --- [http-nio-0.0.0.0-8080-exec-1,f4f81cdba2d94ad1,41e4dcf0bb8a] .o.o.s.o.i.LocalObjectStorageServiceImpl : Delete object object, objectId=8 2023-11-30 16:12:57.284 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:12:57.290 INFO 12 --- [http-nio-0.0.0.0-8080-exec-1,f4f81cdba2d94ad1,41e4dcf0bb8a] c.o.o.s.o.operator.ObjectBlockOperator : deleteByFileId, deletedBlockCount=1 2023-11-30 16:12:57.309 INFO 12 --- [http-nio-0.0.0.0-8080-exec-1,f4f81cdba2d94ad1,41e4dcf0bb8a] c.o.ocp.service.audit.AuditEventService : begin create audit event, param:CreateAuditEventParam(operationEvent=DELETE_SOFTWARE_PACKAGE, eventType=SOFTWARE_PACKAGE_MANAGEMENT, methodSignature=com.oceanbase.ocp.server.controller.SoftwarePackageController.deletePackage, resourceType=SOFTWARE_PACKAGE, resourceId=4, resourceName=oceanbase-ce-libs-4.2.1.1-101010012023111012.el7.x86_64.rpm, parentResource=null, operatorId=100, operatorName=admin, status=SUCCESS, failedReason=null, taskId=null, eventInfo={"id":4}, executionType=NEW, requestUrl=http://10.38.66.53:8080/api/v2/software-packages/4, clientIp=10.38.68.74, serverIp=10.38.66.53, startTime=2023-11-30T16:12:57.273+08:00, endTime=2023-11-30T16:12:57.309+08:00) 2023-11-30 16:12:57.324 INFO 12 --- [http-nio-0.0.0.0-8080-exec-1,f4f81cdba2d94ad1,41e4dcf0bb8a] c.o.o.server.trace.RequestTracingAspect : API OK: [DELETE /api/v2/software-packages/4 client=10.38.68.74, traceId=f4f81cdba2d94ad1, duration=97 ms] 2023-11-30 16:12:57.532 INFO 12 --- [http-nio-0.0.0.0-8080-exec-5,b5e3ead7f4fe4b7b,9d9854f59b4e] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/software-packages/filterMetas?null, client=10.38.68.74, traceId=b5e3ead7f4fe4b7b, method=SuccessResponse com.oceanbase.ocp.server.controller.SoftwarePackageController.getFilterMetas(), args=] 2023-11-30 16:12:57.557 INFO 12 --- [http-nio-0.0.0.0-8080-exec-5,b5e3ead7f4fe4b7b,9d9854f59b4e] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/software-packages/filterMetas client=10.38.68.74, traceId=b5e3ead7f4fe4b7b, duration=26 ms] 2023-11-30 16:12:57.643 INFO 12 --- [http-nio-0.0.0.0-8080-exec-3,4199ff5c5ff64243,1cb9814b819c] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/software-packages?page=1&size=10, client=10.38.68.74, traceId=4199ff5c5ff64243, method=PaginatedResponse com.oceanbase.ocp.server.controller.SoftwarePackageController.listPackages(String,List,List,List,List,boolean,Pageable), args=true,Page request [number: 0, size 10, sort: id: DESC],] 2023-11-30 16:12:57.676 INFO 12 --- [http-nio-0.0.0.0-8080-exec-3,4199ff5c5ff64243,1cb9814b819c] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/software-packages client=10.38.68.74, traceId=4199ff5c5ff64243, duration=35 ms] 2023-11-30 16:13:00.000 INFO 12 --- [pool-ocp-schedules-7,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-11-30 16:13:00.000 INFO 12 --- [pool-ocp-schedules-7,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1701331980 2023-11-30 16:13:00.003 INFO 12 --- [pool-ocp-schedules-12,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-11-30 16:13:00.003 INFO 12 --- [pool-ocp-schedules-12,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-11-30 16:13:00.043 INFO 12 --- [pool-ocp-schedules-12,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-11-30 16:13:00.085 INFO 12 --- [pool-ocp-schedules-8,,0a7566c3a1ea] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-11-30 16:13:00.094 INFO 12 --- [pool-ocp-schedules-8,,0a7566c3a1ea] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-11-30 16:13:00.265 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:12:00.0, nextRun=2023-11-30 16:13:00.0, currentDate=Thu Nov 30 16:13:00 CST 2023 2023-11-30 16:13:00.265 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-30 16:12:00.0, nextRunTime=2023-11-30 16:13:00.0} 2023-11-30 16:13:00.281 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-11-30 16:13:00.281 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:13:00.281 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-11-30 16:13:00.285 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:13:00.286 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:13:00.297 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:12:00.0, nextRun=2023-11-30 16:13:00.0, currentDate=Thu Nov 30 16:13:00 CST 2023 2023-11-30 16:13:00.298 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-30 16:12:00.0, nextRunTime=2023-11-30 16:13:00.0} 2023-11-30 16:13:00.315 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-11-30 16:13:00.315 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:13:00.315 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-11-30 16:13:00.323 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:12:00.0, nextRun=2023-11-30 16:13:00.0, currentDate=Thu Nov 30 16:13:00 CST 2023 2023-11-30 16:13:00.323 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-30 16:12:00.0, nextRunTime=2023-11-30 16:13:00.0} 2023-11-30 16:13:00.336 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-11-30 16:13:00.336 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:13:00.336 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-11-30 16:13:00.344 INFO 12 --- [pool-ocp-schedules-4,,40c18730f0c2] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1368,"minuteWriteQueue":0,"minuteReadCacheSize":7548,"secondWriteCacheSize":586,"seriesIdKeySize":6161,"secondWriteQueue":0,"minuteWriteCacheSize":6473} 2023-11-30 16:13:00.344 INFO 12 --- [pool-ocp-schedules-4,,40c18730f0c2] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=6, tcp_retrans=13, ob_unit_num=6, go_memstats_mspan_sys_bytes=8, go_memstats_stack_sys_bytes=8, exporter_export_metrics_bytes_total=16, node_filesystem_size_bytes=34, http_server_requests_seconds_count=87, ob_system_event_total_waits=16, monagent_input_sql_duration_seconds_sum=41, observer_fd_count=6, processor_process_metrics_total=13, ob_plan_cache_hit_total=10, node_filesystem_readonly=32, go_memstats_frees_total=8, sql_audit_input_collect_data_jumped_total=6, io_await=33, host_available=4, go_memstats_mallocs_total=8, ob_unit_config_max_memory_bytes=12, io_util=33, ob_system_event_detail_total_timeouts=802, ob_tenant_disk_v4_data_size=10, ob_latch_sleep_total=153, ob_tenant_unit_min_memory=13, ob_memstore_total_bytes=10, process_resident_memory_bytes=8, ob_cache_size_bytes=19, ob_tenant_assigned_cpu_total=10, ob_tenant_unit_max_memory=13, input_collect_metrics_total=18, ob_tenant_server_available_disk_free=11, node_disk_io_time_weighted_seconds_total=20, node_disk_write_time_seconds_total=20, cpu_count=13, output_write_metrics_total=7, ob_cluster_sync=6, process_open_fds=8, ob_system_event_detail_time_waited_us=802, process_max_fds=8, ob_server_resource_cpu_assigned_percent=6, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=25, go_memstats_heap_inuse_bytes=8, go_memstats_mcache_sys_bytes=8, ob_tenant_disk_v4_log_size=10, node_memory_memfree_bytes=12, process_exists=18, go_gc_duration_seconds=13, ob_server_resource_disk_bytes=6, ocp_alarm_detect_duration_ms_sum=82, node_memory_cached_bytes=12, ob_table_num=10, ob_disk_free_bytes=6, node_filesystem_avail_bytes=34, ob_unit_config_max_iops=12, node_network_transmit_errs_total=16, ob_plan_cache_memory_bytes=10, go_memstats_last_gc_time_seconds=8, go_gc_duration_seconds_count=8, monitor_exporter_avaliable=17, node_disk_read_bytes_total=20, clock_diff_milliseconds=4, ob_compaction_is_suspended=10, mysql_output_write_sql_milliseconds_summary_count=13, ob_tenant_unit_log_disk_size=10, go_memstats_lookups_total=8, mysql_output_write_sql_count=13, log_tailer_count=8, host_agent_process_status=8, ob_server_resource_cpu=6, go_memstats_alloc_bytes=8, ob_server_num=9, node_cpu_seconds_total_native=18, ob_session_all_num=10, ob_tenant_resource_min_memory=10, ob_memstore_freeze_times=10, ob_system_event_time_waited=16, ob_tenant_resource_min_cpu=10, ob_cluster_status_check=7, monagent_plugin_execute_seconds_total=29, node_net_bandwidth_bps=17, monagent_oceanbase_db_pool_idle=1, go_memstats_heap_idle_bytes=8, go_memstats_gc_sys_bytes=8, ob_server_resource_memory_bytes=6, ob_latch_get_total=153, go_info=9, ob_tenant_assigned_mem_assigned=10, log_tailer_processed_file_count=6, node_load15=12, ob_tenant_server_required_size=13, ob_unit_config_min_iops=12, ob_tenant_unit_max_cpu=13, ob_latch_wait_seconds_total=153, node_memory_buffers_bytes=12, go_memstats_heap_released_bytes=8, node_network_receive_bytes_total=16, ob_memstore_freeze_trigger_bytes=10, ob_waitevent_wait_seconds_total=10, ob_server_stopped_duration_seconds=7, go_memstats_next_gc_bytes=8, ob_session_active_num=10, node_filesystem_files_free=42, node_disk_read_time_seconds_total=20, ob_compaction_frozen_version=10, ob_tenant_unit_log_disk_in_use=10, process_virtual_memory_bytes=8, ob_tenant_assigned_cpu_assigned=10, ob_tenant_unit_min_cpu=13, ob_unit_config_max_cpu=12, http_request_milliseconds_summary_count=28, monagent_oceanbase_audit_timeinfo_cache_size=1, monagent_input_sql_duration_seconds_bucket=51, ob_latch_immediate_get_total=153, node_cpu_seconds_total=20, http_request_milliseconds_summary=31, ob_latch_immediate_miss_total=153, ob_index_error_num=6, ob_tenant_server_available_disk_total=11, ob_latch_spin_get_total=153, ob_tenant_memory_hold_bytes=6, go_memstats_buck_hash_sys_bytes=8, monagent_pipeline_report_metrics_total=9, mysql_output_write_sql_milliseconds_summary_sum=13, ob_tenant_disk_total_size=8, mysql_output_write_task_size=13, ob_tenant_context_memory_count=235, node_disk_reads_completed_total=20, exporter_export_metrics_total=16, ob_sysstat=108, node_ntp_offset_seconds=13, ob_latch_miss_total=153, go_memstats_heap_sys_bytes=8, ocp_meta_db_health=4, ob_disk_total_bytes=6, go_memstats_heap_objects=8, go_memstats_mcache_inuse_bytes=8, go_memstats_other_sys_bytes=8, go_threads=8, http_request_milliseconds_summary_sum=28, http_server_requests_seconds_sum=87, ob_memstore_active_bytes=10, go_memstats_alloc_bytes_total=8, ob_tenant_assigned_mem_total=10, ob_tenant_server_partition_replica_count=13, node_memory_memtotal_bytes=12, node_network_transmit_drop_total=16, ob_system_event_detail_total_waits=802, ob_waitevent_wait_total=10, ob_tenant500_memory_used_bytes=6, node_network_transmit_bytes_total=16, ob_compaction_is_error=10, node_network_receive_drop_total=16, monagent_pipeline_execute_total=16, node_filesystem_files=42, process_start_time_seconds=8, go_memstats_heap_alloc_bytes=8, ob_tenant_resource_max_cpu=10, go_gc_duration_seconds_sum=8, go_memstats_mspan_inuse_bytes=8, mysql_output_write_sql_milliseconds_summary=16, ob_server_resource_memory_assigned_percent=6, monagent_pipeline_execute_seconds_total=16, go_memstats_sys_bytes=8, node_network_receive_errs_total=16, observer_ulimit_max_fd_count=6, ob_plan_cache_access_total=10, ob_tenant_memory_used_bytes=6, ob_tenant_server_row_count=13, monagent_oceanbase_plan_cache_size=1, ob_tenant_server_data_size=13, go_memstats_gc_cpu_fraction=8, monagent_oceanbase_audit_cache_size=1, ob_tenant_context_memory_used_bytes=235, node_disk_writes_completed_total=20, process_cpu_seconds_total=8, node_load1=12, ob_tenant_resource_max_memory=10, go_memstats_stack_inuse_bytes=8, node_load5=12, monagent_input_sql_duration_seconds_count=41, node_disk_written_bytes_total=20, monagent_oceanbase_ash_cache_size=1, ob_tenant_context_memory_hold_bytes=235, go_goroutines=8, ob_compaction_last_version=10, ob_server_resource_cpu_assigned=6, ob_unit_config_min_cpu=12, process_virtual_memory_max_bytes=8, monagent_plugin_execute_total=29, monagent_oceanbase_plan_timeinfo_cache_size=1, ob_tenant500_memory_hold_bytes=6, ob_compaction_current_timestamp=10, oceanbase_connectivity=7} 2023-11-30 16:13:00.348 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:12:31.0, nextRun=2023-11-30 16:13:00.0, currentDate=Thu Nov 30 16:13:00 CST 2023 2023-11-30 16:13:00.348 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-11-30 16:12:31.0, nextRunTime=2023-11-30 16:13:00.0} 2023-11-30 16:13:00.354 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-11-30 16:13:00.354 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:13:00.354 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-11-30 16:13:00.368 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:12:00.0, nextRun=2023-11-30 16:13:00.0, currentDate=Thu Nov 30 16:13:00 CST 2023 2023-11-30 16:13:00.368 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=38, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-30 16:12:00.0, nextRunTime=2023-11-30 16:13:00.0} 2023-11-30 16:13:00.373 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-11-30 16:13:00.373 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:13:00.373 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=38, state=RUNNING, operation=EXECUTE} 2023-11-30 16:13:00.632 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3143, operation:EXECUTE, state: RUNNING 2023-11-30 16:13:00.635 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3143, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-11-30T16:13:00.624+08:00, endTime=null} 2023-11-30 16:13:00.660 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3144, operation:EXECUTE, state: RUNNING 2023-11-30 16:13:00.663 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3144, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-11-30T16:13:00.654+08:00, endTime=null} 2023-11-30 16:13:00.664 INFO 12 --- [alarm-task-1,,89b05ccdc1eb] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-30 16:13:00.689 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3145, operation:EXECUTE, state: RUNNING 2023-11-30 16:13:00.694 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3145, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-11-30T16:13:00.684+08:00, endTime=null} 2023-11-30 16:13:00.705 INFO 12 --- [pool-subtask-executor29,8984c9bcc2be4c9c,d1cdaf2e5f87] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3143, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=3096, task_operation=execute, latest_execution_start_time=2023-11-30T16:13:00.624+08:00, sub_task_instance_id=3143}, listMap={}}, executor=10.38.66.53 2023-11-30 16:13:00.714 INFO 12 --- [pool-subtask-executor30,8984c9bcc2be4c9c,7c6cce1b713c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3144, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=3097, task_operation=execute, latest_execution_start_time=2023-11-30T16:13:00.653+08:00, sub_task_instance_id=3144}, listMap={}}, executor=10.38.66.53 2023-11-30 16:13:00.757 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3146, operation:EXECUTE, state: RUNNING 2023-11-30 16:13:00.759 INFO 12 --- [alarm-task-2,,8ed9b273a5a2] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-30T16:13:00.755330+08:00, lastProcessTime=2023-11-30T16:12:50.685447+08:00 2023-11-30 16:13:00.767 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3146, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-11-30T16:13:00.735+08:00, endTime=null} 2023-11-30 16:13:00.809 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,bea79e5907e6] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3145, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=3098, task_operation=execute, latest_execution_start_time=2023-11-30T16:13:00.683+08:00, sub_task_instance_id=3145}, listMap={}}, executor=10.38.66.53 2023-11-30 16:13:00.822 INFO 12 --- [pool-subtask-executor30,8984c9bcc2be4c9c,7c6cce1b713c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster local_v4, trans stat result is empty. 2023-11-30 16:13:00.844 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,8c4c2e7caf8b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3146, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=3099, task_operation=execute, latest_execution_start_time=2023-11-30T16:13:00.734+08:00, sub_task_instance_id=3146}, listMap={}}, executor=10.38.66.53 2023-11-30 16:13:00.845 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,8c4c2e7caf8b] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-11-30 16:13:00.845 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,bea79e5907e6] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster local_v4, sql count is 0. 2023-11-30 16:13:00.845 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,bea79e5907e6] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster local_v4, result is empty. 2023-11-30 16:13:00.849 INFO 12 --- [pool-subtask-executor30,8984c9bcc2be4c9c,7c6cce1b713c] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3144, operation:EXECUTE, state: SUCCESSFUL 2023-11-30 16:13:00.849 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3147, operation:EXECUTE, state: RUNNING 2023-11-30 16:13:00.857 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3147, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-11-30T16:13:00.811+08:00, endTime=null} 2023-11-30 16:13:00.859 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,bea79e5907e6] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3145, operation:EXECUTE, state: SUCCESSFUL 2023-11-30 16:13:00.894 INFO 12 --- [alarm-detect-14,,4c94f845bf58] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1701331960 ~ 1701331970 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.607, values={1701331970=1.607}, valueCount=1, labels={host=10.38.68.74, host_name=az-rta-prod-database-database-mysql-13, ob_cluster_id=1, ob_cluster_name=local_v4, obzone=zone1, svr_ip=10.38.68.74})]}, matched=true, message=null, level=3) 2023-11-30 16:13:00.894 INFO 12 --- [alarm-detect-14,,4c94f845bf58] .o.o.s.a.c.d.AlarmRuleDetectStateManager : alarmType ob_host_load1_per_cpu_over_threshold targets.size 1 was marked as not match 2023-11-30 16:13:00.915 INFO 12 --- [pool-48890-thread-2,8984c9bcc2be4c9c,feb217655244] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:local_v4, tenant:sys 2023-11-30 16:13:00.916 INFO 12 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-11-30 16:13:00.917 INFO 12 --- [pool-48890-thread-1,8984c9bcc2be4c9c,fc746e0b5dfa] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:local_v4, tenant:ocp 2023-11-30 16:13:00.937 INFO 12 --- [alarm-task-2,,8ed9b273a5a2] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-30T16:13:00.755330+08:00 2023-11-30 16:13:00.971 INFO 12 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-30 16:13:00.976 INFO 12 --- [pool-subtask-executor29,8984c9bcc2be4c9c,d1cdaf2e5f87] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3143, operation:EXECUTE, state: SUCCESSFUL 2023-11-30 16:13:00.983 INFO 12 --- [pool-48890-thread-1,8984c9bcc2be4c9c,fc746e0b5dfa] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:00.986 INFO 12 --- [pool-48890-thread-1,8984c9bcc2be4c9c,fc746e0b5dfa] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:00.986 INFO 12 --- [pool-48890-thread-1,8984c9bcc2be4c9c,fc746e0b5dfa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:00.988 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3097, status:SUCCESS 2023-11-30 16:13:00.989 INFO 12 --- [pool-48890-thread-1,8984c9bcc2be4c9c,fc746e0b5dfa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:13:00.993 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3097, status:SUCCESS 2023-11-30 16:13:00.997 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,94c1fa8ee734] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3147, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=3100, task_operation=execute, latest_execution_start_time=2023-11-30T16:13:00.803+08:00, sub_task_instance_id=3147}, listMap={}}, executor=10.38.66.53 2023-11-30 16:13:01.003 INFO 12 --- [pool-48890-thread-1,8984c9bcc2be4c9c,fc746e0b5dfa] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:01.003 INFO 12 --- [pool-48890-thread-1,8984c9bcc2be4c9c,fc746e0b5dfa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:01.007 INFO 12 --- [pool-48890-thread-1,8984c9bcc2be4c9c,fc746e0b5dfa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-11-30 16:13:01.012 INFO 12 --- [pool-48890-thread-2,8984c9bcc2be4c9c,feb217655244] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:01.027 INFO 12 --- [pool-48890-thread-2,8984c9bcc2be4c9c,feb217655244] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:01.027 INFO 12 --- [pool-48890-thread-2,8984c9bcc2be4c9c,feb217655244] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:01.028 INFO 12 --- [pool-48890-thread-1,8984c9bcc2be4c9c,fc746e0b5dfa] o.i.t.TenantCompactionCollectServiceImpl : Cluster never major compaction, just skip 2023-11-30 16:13:01.030 INFO 12 --- [pool-48890-thread-2,8984c9bcc2be4c9c,feb217655244] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:13:01.036 INFO 12 --- [pool-48890-thread-2,8984c9bcc2be4c9c,feb217655244] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:01.036 INFO 12 --- [pool-48890-thread-2,8984c9bcc2be4c9c,feb217655244] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:01.037 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3098, status:SUCCESS 2023-11-30 16:13:01.039 INFO 12 --- [pool-48890-thread-2,8984c9bcc2be4c9c,feb217655244] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-11-30 16:13:01.040 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3098, status:SUCCESS 2023-11-30 16:13:01.047 INFO 12 --- [pool-48890-thread-2,8984c9bcc2be4c9c,feb217655244] o.i.t.TenantCompactionCollectServiceImpl : Cluster never major compaction, just skip 2023-11-30 16:13:01.052 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,8c4c2e7caf8b] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-11-30 16:13:01.064 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,8c4c2e7caf8b] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3146, operation:EXECUTE, state: SUCCESSFUL 2023-11-30 16:13:01.073 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,94c1fa8ee734] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3147, operation:EXECUTE, state: SUCCESSFUL 2023-11-30 16:13:01.190 INFO 12 --- [ForkJoinPool-11-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.38.68.74:62889/api/v1/log/alarms, params:null 2023-11-30 16:13:01.355 INFO 12 --- [pool-overview-container-5,,316e0120432b] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=3, alarmHistory size=6 2023-11-30 16:13:01.549 INFO 12 --- [pool-prometheus-minute-schedule-2,,20ec46311a06] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1701332040, interval=60 2023-11-30 16:13:01.550 INFO 12 --- [pool-prometheus-minute-schedule-1,,d4ab60abbeab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1701332040, interval=60 2023-11-30 16:13:01.577 INFO 12 --- [pool-prometheus-minute-schedule-2,,20ec46311a06] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=1, collectTaskCount=1 2023-11-30 16:13:01.578 INFO 12 --- [pool-prometheus-minute-schedule-2,,20ec46311a06] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1701331981, count=1, elapsed=1699630649597 2023-11-30 16:13:01.618 INFO 12 --- [pool-prometheus-minute-schedule-1,,d4ab60abbeab] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2023-11-30 16:13:01.618 INFO 12 --- [pool-prometheus-minute-schedule-1,,d4ab60abbeab] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1701331981, count=2, elapsed=1699630649637 2023-11-30 16:13:01.671 INFO 12 --- [pool-ocp-schedules-2,,9af79636fd85] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-30 16:13:02.103 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3096, status:SUCCESS 2023-11-30 16:13:02.106 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3096, status:SUCCESS 2023-11-30 16:13:02.135 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3099, status:SUCCESS 2023-11-30 16:13:02.140 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3099, status:SUCCESS 2023-11-30 16:13:02.158 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3100, status:SUCCESS 2023-11-30 16:13:02.160 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3100, status:SUCCESS 2023-11-30 16:13:02.177 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.38.66.53:8080 2023-11-30 16:13:02.343 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.38.66.53:8080 2023-11-30 16:13:02.351 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.38.66.53:8080 2023-11-30 16:13:02.382 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.38.66.53:8080 2023-11-30 16:13:03.287 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:13:03.288 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:13:03.724 INFO 12 --- [pool-ocp-schedules-9,,0e99f43704dc] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=586751257718591488 2023-11-30 16:13:05.116 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.38.66.53:8080 2023-11-30 16:13:05.160 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.38.66.53:8080 2023-11-30 16:13:05.227 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.38.66.53:8080 2023-11-30 16:13:05.237 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.38.66.53:8080 2023-11-30 16:13:05.598 INFO 12 --- [pool-ocp-schedules-14,,526d2a8ae462] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-11-30 16:13:05.602 INFO 12 --- [pool-ocp-schedules-14,,526d2a8ae462] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=1, successCount=0 2023-11-30 16:13:05.668 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:05.670 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:05.670 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:05.673 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:13:05.678 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:05.678 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:05.680 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-11-30 16:13:05.770 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:12:06.0, nextRun=2023-11-30 16:13:05.0, currentDate=Thu Nov 30 16:13:05 CST 2023 2023-11-30 16:13:05.770 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-11-30 16:12:06.0, nextRunTime=2023-11-30 16:13:05.0} 2023-11-30 16:13:05.775 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-11-30 16:13:05.775 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:13:05.775 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-11-30 16:13:05.815 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=local_v4, tenantName=sys, dbUser=ocp_monitor 2023-11-30 16:13:05.819 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:05.821 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:05.821 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:05.824 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:13:05.830 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:05.830 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:05.833 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-11-30 16:13:05.839 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:05.839 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:05.842 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-11-30 16:13:05.845 INFO 12 --- [ForkJoinPool-2064-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.38.68.74:2892, status=ACTIVE, startServiceTime=1701328610966002, active=true 2023-11-30 16:13:05.846 INFO 12 --- [ForkJoinPool-2064-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.38.68.74:2892, stopTime=0, stopped=false 2023-11-30 16:13:05.846 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-30 16:13:05.846 INFO 12 --- [pool-ocp-schedules-5,,2b0c480ea081] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-11-30 16:13:05.851 INFO 12 --- [pool-ocp-schedules-18,,2c1d4b1bc419] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-11-30 16:13:05.856 INFO 12 --- [pool-ocp-schedules-18,,2c1d4b1bc419] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=1, successCount=0 2023-11-30 16:13:05.858 INFO 12 --- [pool-ocp-schedules-5,,2b0c480ea081] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=1, successCount=0 2023-11-30 16:13:05.880 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-30 16:13:05.880 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.38.68.74, port=2892, sqlPort=2891, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T15:16:51+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.38.68.74, port=2892, sqlPort=2891, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T07:16:50.966002Z, stopTime=1970-01-01T00:00Z} 2023-11-30 16:13:05.910 INFO 12 --- [ForkJoinPool-2065-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:05.912 INFO 12 --- [ForkJoinPool-2065-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:05.913 INFO 12 --- [ForkJoinPool-2065-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:05.915 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.38.68.74, port=2892, sqlPort=2891, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T07:16:50.966002Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-30 16:13:05.915 INFO 12 --- [ForkJoinPool-2065-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:13:05.919 INFO 12 --- [ForkJoinPool-2065-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:05.919 INFO 12 --- [ForkJoinPool-2065-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:05.921 INFO 12 --- [ForkJoinPool-2065-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:13:05.924 INFO 12 --- [ForkJoinPool-2065-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2023-11-30 16:13:05.938 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=local_v4, tenantName=sys, dbUser=ocp_monitor 2023-11-30 16:13:05.944 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:05.946 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:05.946 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:05.949 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:13:05.954 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3148, operation:EXECUTE, state: RUNNING 2023-11-30 16:13:05.956 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:05.956 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:05.958 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3148, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-11-30T16:13:05.950+08:00, endTime=null} 2023-11-30 16:13:05.962 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:13:05.983 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=local_v4, obClusterId=1 2023-11-30 16:13:05.983 INFO 12 --- [ForkJoinPool-2064-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=local_v4, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"local_v4","ObClusterId":1,"ObRegion":"local_v4","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.38.68.74:2892","role":"LEADER","sql_port":2891}],"Type":"PRIMARY","timestamp":1701331607673000}, createTime=2023-11-30T15:22:21+08:00, creator=admin, obVersion=4.2.1.1, tenantCount=2) 2023-11-30 16:13:06.009 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,c40e8cfc125a] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3148, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=3101, task_operation=execute, latest_execution_start_time=2023-11-30T16:13:05.950+08:00, sub_task_instance_id=3148}, listMap={}}, executor=10.38.66.53 2023-11-30 16:13:06.028 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,c40e8cfc125a] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3148, operation:EXECUTE, state: SUCCESSFUL 2023-11-30 16:13:06.288 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:13:06.290 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:13:06.296 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3101, status:SUCCESS 2023-11-30 16:13:06.298 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3101, status:SUCCESS 2023-11-30 16:13:07.119 INFO 12 --- [pool-ocp-schedules-11,,e176b573a3bf] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-11-30 16:13:07.119 INFO 12 --- [pool-Check host status13,,d586bd24344d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.38.68.74, originalStatus=ONLINE 2023-11-30 16:13:07.126 INFO 12 --- [pool-Check host status13,,d586bd24344d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-30 16:13:07.178 INFO 12 --- [pool-Check host status13,,d586bd24344d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.38.68.74:62888/api/v1/time, params:null 2023-11-30 16:13:07.182 INFO 12 --- [pool-Check host status13,,d586bd24344d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=2, ip=10.38.68.74, timestamp=2023-11-30T16:13:07.190199600+08:00 2023-11-30 16:13:07.182 INFO 12 --- [pool-Check host status13,,d586bd24344d] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-11-30T16:13:07.182+08:00 2023-11-30 16:13:07.188 INFO 12 --- [pool-Check host status13,,d586bd24344d] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-11-30T16:13:07.182+08:00 and status AVAILABLE 2023-11-30 16:13:07.202 INFO 12 --- [pool-Check host status13,,d586bd24344d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.38.68.74], timeoutMillis=20000 2023-11-30 16:13:07.374 INFO 12 --- [http-nio-0.0.0.0-8080-exec-4,,] c.o.o.c.s.util.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/software-packages, client 10.38.68.74, traceId ] 2023-11-30 16:13:07.387 INFO 12 --- [http-nio-0.0.0.0-8080-exec-4,c708969b08464027,10c9ca14fe67] c.o.o.server.trace.RequestTracingAspect : API: [POST /api/v2/software-packages?null, client=10.38.68.74, traceId=c708969b08464027, method=SuccessResponse com.oceanbase.ocp.server.controller.SoftwarePackageController.uploadPackage(MultipartFile), args=] 2023-11-30 16:13:07.447 INFO 12 --- [http-nio-0.0.0.0-8080-exec-4,c708969b08464027,10c9ca14fe67] .o.o.s.o.i.LocalObjectStorageServiceImpl : Init saving object, creatorId=100, bucket=sys-package, name=oceanbase-ce-libs-4.2.1.1-101010012023111012.el7.x86_64.rpm, objectId=9 2023-11-30 16:13:07.547 INFO 12 --- [http-nio-0.0.0.0-8080-exec-4,c708969b08464027,10c9ca14fe67] .o.o.s.o.i.LocalObjectStorageServiceImpl : Finish saving object, objectId=9, name=oceanbase-ce-libs-4.2.1.1-101010012023111012.el7.x86_64.rpm 2023-11-30 16:13:07.547 INFO 12 --- [http-nio-0.0.0.0-8080-exec-4,c708969b08464027,10c9ca14fe67] c.o.o.s.s.SoftwarePackageServiceImpl : Save software object, name=oceanbase-ce-libs-4.2.1.1-101010012023111012.el7.x86_64.rpm, objectId=9 2023-11-30 16:13:07.558 INFO 12 --- [http-nio-0.0.0.0-8080-exec-4,c708969b08464027,10c9ca14fe67] c.o.o.s.s.SoftwarePackageServiceImpl : Save software success, name=oceanbase-ce-libs-4.2.1.1-101010012023111012.el7.x86_64.rpm, objectId=9, softwareId=6 2023-11-30 16:13:07.558 INFO 12 --- [http-nio-0.0.0.0-8080-exec-4,c708969b08464027,10c9ca14fe67] c.o.o.server.trace.RequestTracingAspect : API OK: [POST /api/v2/software-packages client=10.38.68.74, traceId=c708969b08464027, duration=172 ms] 2023-11-30 16:13:07.565 WARN 12 --- [http-nio-0.0.0.0-8080-exec-4,c708969b08464027,10c9ca14fe67] o.h.p.entity.AbstractEntityPersister : HHH000502: The [objectMeta] property of the [com.oceanbase.ocp.core.metadb.repo.compute.softwarepackage.SoftwarePackageMetaEntity] entity was modified, but it won't be updated because the property is immutable. 2023-11-30 16:13:07.636 INFO 12 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2023-11-30 16:13:07.689 INFO 12 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:07.690 INFO 12 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:07.691 INFO 12 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:07.693 INFO 12 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:13:07.697 INFO 12 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:07.697 INFO 12 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:07.700 INFO 12 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-11-30 16:13:07.703 INFO 12 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (local_v4:1-1) 2023-11-30 16:13:07.712 INFO 12 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.38.66.53, sql port:2883 2023-11-30 16:13:07.712 INFO 12 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:07.719 INFO 12 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-30 16:13:07.775 INFO 12 --- [http-nio-0.0.0.0-8080-exec-9,f8882571815e4fc7,410198f3f98d] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/software-packages?page=1&size=10, client=10.38.68.74, traceId=f8882571815e4fc7, method=PaginatedResponse com.oceanbase.ocp.server.controller.SoftwarePackageController.listPackages(String,List,List,List,List,boolean,Pageable), args=true,Page request [number: 0, size 10, sort: id: DESC],] 2023-11-30 16:13:07.777 INFO 12 --- [http-nio-0.0.0.0-8080-exec-2,27e292a01682424e,4d38e6a04935] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/software-packages/filterMetas?null, client=10.38.68.74, traceId=27e292a01682424e, method=SuccessResponse com.oceanbase.ocp.server.controller.SoftwarePackageController.getFilterMetas(), args=] 2023-11-30 16:13:07.781 INFO 12 --- [http-nio-0.0.0.0-8080-exec-2,27e292a01682424e,4d38e6a04935] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/software-packages/filterMetas client=10.38.68.74, traceId=27e292a01682424e, duration=6 ms] 2023-11-30 16:13:07.818 INFO 12 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=local_v4:1, server=1, level=0->1, c=0) - source=2023-11-30T16:10:44, start=2023-11-30T16:07:37, end=2023-11-30T16:10:07, updates=81, granule=120000ms, delay=187000ms 2023-11-30 16:13:07.835 INFO 12 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=local_v4:1, server=1, level=0->1, c=1) - not enough cached interval (source=2023-11-30T16:10:44, target=2023-11-30T16:10:07, granule=120000ms, delay=37000ms) 2023-11-30 16:13:07.835 INFO 12 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=local_v4:1, server=1, level=1->2, c=0) - not enough cached interval (source=2023-11-30T16:10:07, target=2023-11-30T16:01:07, granule=600000ms, delay=540000ms) 2023-11-30 16:13:07.835 INFO 12 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=199ms, period=120000ms 2023-11-30 16:13:07.844 INFO 12 --- [http-nio-0.0.0.0-8080-exec-9,f8882571815e4fc7,410198f3f98d] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/software-packages client=10.38.68.74, traceId=f8882571815e4fc7, duration=70 ms] 2023-11-30 16:13:08.354 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.38.66.53:8080 2023-11-30 16:13:09.290 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:13:09.291 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:13:09.518 INFO 12 --- [pool-ocp-schedules-10,,17e816262f6a] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[997e1fae-052d-4a24-a32c-2dd49f574f28, a1cfb9f7-f190-4ebe-9f7b-70b8f35ffcf1] 2023-11-30 16:13:09.518 INFO 12 --- [pool-ocp-schedules-10,,17e816262f6a] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[997e1fae-052d-4a24-a32c-2dd49f574f28, a1cfb9f7-f190-4ebe-9f7b-70b8f35ffcf1] 2023-11-30 16:13:10.663 INFO 12 --- [alarm-task-1,,89b05ccdc1eb] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-30 16:13:10.682 INFO 12 --- [alarm-task-2,,8ed9b273a5a2] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-30T16:13:10.681032+08:00, lastProcessTime=2023-11-30T16:13:00.755330+08:00 2023-11-30 16:13:10.759 INFO 12 --- [alarm-detect-15,,b6fbda5b34a6] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1701331970 ~ 1701331980 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.925, values={1701331980=1.925}, valueCount=1, labels={host=10.38.68.74, host_name=az-rta-prod-database-database-mysql-13, ob_cluster_id=1, ob_cluster_name=local_v4, obzone=zone1, svr_ip=10.38.68.74})]}, matched=true, message=null, level=3) 2023-11-30 16:13:10.760 INFO 12 --- [alarm-detect-15,,b6fbda5b34a6] .o.o.s.a.c.d.AlarmRuleDetectStateManager : alarmType ob_host_load1_per_cpu_over_threshold targets.size 1 was marked as not match 2023-11-30 16:13:10.783 INFO 12 --- [alarm-task-2,,8ed9b273a5a2] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-30T16:13:10.681032+08:00 2023-11-30 16:13:11.358 INFO 12 --- [pool-overview-container-6,,bb6e6806d641] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=3, alarmHistory size=6 2023-11-30 16:13:11.715 INFO 12 --- [pool-ocp-schedules-6,,3ed218040e33] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-30 16:13:12.186 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.38.66.53:8080 2023-11-30 16:13:12.292 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:13:12.298 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:13:12.347 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.38.66.53:8080 2023-11-30 16:13:12.354 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.38.66.53:8080 2023-11-30 16:13:12.389 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.38.66.53:8080 2023-11-30 16:13:13.249 INFO 12 --- [pool-Check host status13,,d586bd24344d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1701331993 10 9 , error= 2023-11-30 16:13:13.252 INFO 12 --- [pool-Check host status13,,d586bd24344d] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-11-30T16:13:13.249+08:00 2023-11-30 16:13:13.256 INFO 12 --- [pool-Check host status13,,d586bd24344d] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-11-30T16:13:13.249+08:00 and status AVAILABLE 2023-11-30 16:13:13.262 INFO 12 --- [pool-Check host status13,,d586bd24344d] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=2 2023-11-30 16:13:13.301 INFO 12 --- [pool-Check host status13,,d586bd24344d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-30 16:13:13.349 INFO 12 --- [pool-Check host status13,,d586bd24344d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.38.68.74:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-11-30 16:13:13.410 INFO 12 --- [pool-Check host status13,,d586bd24344d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2023-11-30 16:13:13.435 INFO 12 --- [pool-Check host status13,,d586bd24344d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2023-11-30 16:13:13.502 INFO 12 --- [pool-Check host status13,,d586bd24344d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.38.68.74:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-11-30 16:13:13.565 INFO 12 --- [pool-Check host status13,,d586bd24344d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2023-11-30 16:13:13.588 INFO 12 --- [pool-Check host status13,,d586bd24344d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2023-11-30 16:13:13.632 INFO 12 --- [pool-Check host status13,,d586bd24344d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.38.68.74:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-11-30 16:13:13.694 INFO 12 --- [pool-Check host status13,,d586bd24344d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2023-11-30 16:13:13.710 INFO 12 --- [pool-Check host status13,,d586bd24344d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2023-11-30 16:13:13.740 INFO 12 --- [pool-Check host status14,,1e9ef198fa46] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.38.68.74, originalStatus=ONLINE 2023-11-30 16:13:13.750 INFO 12 --- [pool-Check host status14,,1e9ef198fa46] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-30 16:13:13.799 INFO 12 --- [pool-Check host status14,,1e9ef198fa46] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.38.68.74:62888/api/v1/agent/status, params:null 2023-11-30 16:13:13.807 INFO 12 --- [pool-Check host status14,,1e9ef198fa46] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.38.68.74 2023-11-30 16:13:13.807 INFO 12 --- [pool-Check host status14,,1e9ef198fa46] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.38.68.74 2023-11-30 16:13:13.807 INFO 12 --- [pool-Check host status14,,1e9ef198fa46] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.38.68.74 2023-11-30 16:13:14.156 INFO 12 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1701331920, nodeCount=342, affectRows=342 2023-11-30 16:13:14.158 INFO 12 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1701331920, nodeCount=342, validCount=342 2023-11-30 16:13:15.122 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.38.66.53:8080 2023-11-30 16:13:15.299 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:13:15.301 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:13:18.301 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:13:18.316 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:13:18.846 INFO 12 --- [http-nio-0.0.0.0-8080-exec-8,f1d0b5adfaec4afb,fbcdd41b8f2b] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.38.66.53, traceId=f1d0b5adfaec4afb, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-11-30 16:13:18.850 INFO 12 --- [http-nio-0.0.0.0-8080-exec-8,f1d0b5adfaec4afb,fbcdd41b8f2b] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.38.66.53, traceId=f1d0b5adfaec4afb, duration=4 ms] 2023-11-30 16:13:20.663 INFO 12 --- [alarm-task-1,,89b05ccdc1eb] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-30 16:13:20.690 INFO 12 --- [alarm-task-2,,8ed9b273a5a2] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-30T16:13:20.689660+08:00, lastProcessTime=2023-11-30T16:13:10.681032+08:00 2023-11-30 16:13:20.788 INFO 12 --- [alarm-detect-1,,a27fff96f2cd] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1701331980 ~ 1701331990 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={load1_per_cpu > 2=[ExpressionEvaluateResult(matchValue=2.0, metric=load1_per_cpu, match=true, value=2.45, values={1701331990=2.45}, valueCount=1, labels={host=10.38.68.74, host_name=az-rta-prod-database-database-mysql-13, ob_cluster_id=1, ob_cluster_name=local_v4, obzone=zone1, svr_ip=10.38.68.74})]}, matched=true, message=null, level=2) 2023-11-30 16:13:20.788 INFO 12 --- [alarm-detect-1,,a27fff96f2cd] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1701331980 ~ 1701331990 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=2.45, values={1701331990=2.45}, valueCount=1, labels={host=10.38.68.74, host_name=az-rta-prod-database-database-mysql-13, ob_cluster_id=1, ob_cluster_name=local_v4, obzone=zone1, svr_ip=10.38.68.74})]}, matched=true, message=null, level=3) 2023-11-30 16:13:20.788 INFO 12 --- [alarm-detect-1,,a27fff96f2cd] c.o.o.s.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_host_load1_per_cpu_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.38.68.74-3, alarm_template_id=0:host=10.38.68.74-2] 2023-11-30 16:13:20.789 INFO 12 --- [alarm-task-2,,8ed9b273a5a2] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-30T16:13:20.689660+08:00 2023-11-30 16:13:20.898 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:12:20.0, nextRun=2023-11-30 16:13:20.0, currentDate=Thu Nov 30 16:13:20 CST 2023 2023-11-30 16:13:20.898 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-11-30 16:12:20.0, nextRunTime=2023-11-30 16:13:20.0} 2023-11-30 16:13:20.907 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-11-30 16:13:20.907 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:13:20.907 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-11-30 16:13:21.013 INFO 12 --- [pool-ocp-schedules-3,,3a93348e2f7a] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-11-30 16:13:21.016 INFO 12 --- [pool-ocp-schedules-3,,3a93348e2f7a] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-11-30 16:13:21.122 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3149, operation:EXECUTE, state: RUNNING 2023-11-30 16:13:21.127 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3149, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-11-30T16:13:21.117+08:00, endTime=null} 2023-11-30 16:13:21.167 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,10d2f222a0bb] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3149, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=3102, task_operation=execute, latest_execution_start_time=2023-11-30T16:13:21.117+08:00, sub_task_instance_id=3149}, listMap={}}, executor=10.38.66.53 2023-11-30 16:13:21.170 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,10d2f222a0bb] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=1, successCount=0 2023-11-30 16:13:21.170 INFO 12 --- [ForkJoinPool-2069-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-11-30 16:13:21.179 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,10d2f222a0bb] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3149, operation:EXECUTE, state: SUCCESSFUL 2023-11-30 16:13:21.317 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:13:21.318 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:13:21.354 INFO 12 --- [pool-overview-container-4,,c43570bb243b] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=3, alarmHistory size=6 2023-11-30 16:13:21.692 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3102, status:SUCCESS 2023-11-30 16:13:21.694 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3102, status:SUCCESS 2023-11-30 16:13:21.744 INFO 12 --- [pool-ocp-schedules-5,,40f8cb30cd2d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-30 16:13:22.195 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.38.66.53:8080 2023-11-30 16:13:22.352 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.38.66.53:8080 2023-11-30 16:13:22.358 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.38.66.53:8080 2023-11-30 16:13:22.401 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.38.66.53:8080 2023-11-30 16:13:23.357 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.38.66.53:8080 2023-11-30 16:13:24.321 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:13:24.332 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:13:25.129 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.38.66.53:8080 2023-11-30 16:13:25.222 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:12:25.0, nextRun=2023-11-30 16:13:25.0, currentDate=Thu Nov 30 16:13:25 CST 2023 2023-11-30 16:13:25.223 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-11-30 16:12:25.0, nextRunTime=2023-11-30 16:13:25.0} 2023-11-30 16:13:25.233 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-11-30 16:13:25.233 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:13:25.233 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-11-30 16:13:26.207 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3150, operation:EXECUTE, state: RUNNING 2023-11-30 16:13:26.211 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3150, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-11-30T16:13:26.202+08:00, endTime=null} 2023-11-30 16:13:26.252 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,c898a0b6247b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3150, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=3103, task_operation=execute, latest_execution_start_time=2023-11-30T16:13:26.202+08:00, sub_task_instance_id=3150}, listMap={host_ids=[2]}}, executor=10.38.66.53 2023-11-30 16:13:26.316 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,c898a0b6247b] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=2, ip=10.38.68.74] desired exporter count: 6, actual exporter count: 6 2023-11-30 16:13:26.328 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,c898a0b6247b] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3150, operation:EXECUTE, state: SUCCESSFUL 2023-11-30 16:13:26.835 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3103, status:SUCCESS 2023-11-30 16:13:26.837 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3103, status:SUCCESS 2023-11-30 16:13:27.333 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:13:27.334 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:13:28.262 ERROR 12 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Error in executing SQL_PERF_DIAGNOSER org.springframework.dao.PermissionDeniedDataAccessException: ### Error querying database. Cause: java.sql.SQLInvalidAuthorizationSpecException: (conn=3146158) Statement timeout occurred, please set the variable ob_query_timeout to a larger value an then restart the statement ### The error may exist in com/oceanbase/ocp/perf/sql/diag/monitordb/dao/OcpPerfSqlDiagnosisTaskDao.java (best guess) ### The error may involve com.oceanbase.ocp.perf.sql.diag.monitordb.dao.OcpPerfSqlDiagnosisTaskDao.getLastExecutionForUpdate-Inline ### The error occurred while setting parameters ### SQL: SELECT task_name AS taskName, begin_interval_time AS beginIntervalTime, end_interval_time AS endIntervalTime, executor, create_time as createTime, update_time as updateTime, status FROM ocp_perf_sql_diagnosis_task WHERE task_name = ? ORDER BY end_interval_time DESC LIMIT 1 FOR UPDATE NOWAIT ### Cause: java.sql.SQLInvalidAuthorizationSpecException: (conn=3146158) Statement timeout occurred, please set the variable ob_query_timeout to a larger value an then restart the statement ; (conn=3146158) Statement timeout occurred, please set the variable ob_query_timeout to a larger value an then restart the statement; nested exception is java.sql.SQLInvalidAuthorizationSpecException: (conn=3146158) Statement timeout occurred, please set the variable ob_query_timeout to a larger value an then restart the statement at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:90) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:70) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:79) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:88) ~[mybatis-spring-2.0.4.jar!/:2.0.4] at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:440) ~[mybatis-spring-2.0.4.jar!/:2.0.4] at com.sun.proxy.$Proxy324.selectOne(Unknown Source) ~[na:na] at org.mybatis.spring.SqlSessionTemplate.selectOne(SqlSessionTemplate.java:159) ~[mybatis-spring-2.0.4.jar!/:2.0.4] at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:87) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:152) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:85) ~[mybatis-3.5.6.jar!/:3.5.6] at com.sun.proxy.$Proxy533.getLastExecutionForUpdate(Unknown Source) ~[na:na] at com.oceanbase.ocp.perf.sql.diag.internal.access.OcpPerfSqlDiagnosisTaskDbAccessImpl.getLastExecutionForUpdate(OcpPerfSqlDiagnosisTaskDbAccessImpl.java:40) ~[ocp-perf-sql-diag-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.perf.sql.diag.internal.OcpPerfSqlDiagnosisTaskService.getNextExecution(OcpPerfSqlDiagnosisTaskService.java:40) ~[ocp-perf-sql-diag-pipeline-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.perf.sql.diag.internal.OcpPerfSqlDiagnosisTaskService$$FastClassBySpringCGLIB$$a1a13d9d.invoke() ~[ocp-perf-sql-diag-pipeline-4.0.3-20230301.jar!/:4.0.3-20230301] at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) ~[spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) ~[spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) ~[spring-tx-5.3.23.jar!/:5.3.23] at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) ~[spring-tx-5.3.23.jar!/:5.3.23] at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) ~[spring-tx-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) ~[spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) ~[spring-aop-5.3.23.jar!/:5.3.23] at com.oceanbase.ocp.perf.sql.diag.internal.OcpPerfSqlDiagnosisTaskService$$EnhancerBySpringCGLIB$$e88b9a71.getNextExecution() ~[ocp-perf-sql-diag-pipeline-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.perf.sql.diag.internal.AbstractDiagnoseService.lambda$init$1(AbstractDiagnoseService.java:138) ~[ocp-perf-sql-diag-pipeline-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_312] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[na:1.8.0_312] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_312] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLInvalidAuthorizationSpecException: (conn=3146158) Statement timeout occurred, please set the variable ob_query_timeout to a larger value an then restart the statement at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:114) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:202) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseStatement.executeExceptionEpilogue(OceanBaseStatement.java:315) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:243) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.JDBC4PreparedStatement.execute(JDBC4PreparedStatement.java:161) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidPooledPreparedStatement.execute(DruidPooledPreparedStatement.java:497) ~[druid-1.2.8.jar!/:1.2.8] at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:64) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:79) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:63) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:325) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:89) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:147) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:76) ~[mybatis-3.5.6.jar!/:3.5.6] at sun.reflect.GeneratedMethodAccessor463.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_312] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_312] at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:426) ~[mybatis-spring-2.0.4.jar!/:2.0.4] ... 28 common frames omitted Caused by: com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException: Statement timeout occurred, please set the variable ob_query_timeout to a larger value an then restart the statement at com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException.of(OceanBaseSqlException.java:79) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:199) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:180) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:372) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:234) ~[oceanbase-client-2.4.2.jar!/:na] ... 44 common frames omitted Caused by: java.sql.SQLException: Statement timeout occurred, please set the variable ob_query_timeout to a larger value an then restart the statement at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:2192) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2057) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:1951) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:370) ~[oceanbase-client-2.4.2.jar!/:na] ... 45 common frames omitted 2023-11-30 16:13:28.265 ERROR 12 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Error in executing SQL_INDEX_DIAGNOSER org.springframework.dao.PermissionDeniedDataAccessException: ### Error querying database. Cause: java.sql.SQLInvalidAuthorizationSpecException: (conn=3145736) Statement timeout occurred, please set the variable ob_query_timeout to a larger value an then restart the statement ### The error may exist in com/oceanbase/ocp/perf/sql/diag/monitordb/dao/OcpPerfSqlDiagnosisTaskDao.java (best guess) ### The error may involve com.oceanbase.ocp.perf.sql.diag.monitordb.dao.OcpPerfSqlDiagnosisTaskDao.getLastExecutionForUpdate-Inline ### The error occurred while setting parameters ### SQL: SELECT task_name AS taskName, begin_interval_time AS beginIntervalTime, end_interval_time AS endIntervalTime, executor, create_time as createTime, update_time as updateTime, status FROM ocp_perf_sql_diagnosis_task WHERE task_name = ? ORDER BY end_interval_time DESC LIMIT 1 FOR UPDATE NOWAIT ### Cause: java.sql.SQLInvalidAuthorizationSpecException: (conn=3145736) Statement timeout occurred, please set the variable ob_query_timeout to a larger value an then restart the statement ; (conn=3145736) Statement timeout occurred, please set the variable ob_query_timeout to a larger value an then restart the statement; nested exception is java.sql.SQLInvalidAuthorizationSpecException: (conn=3145736) Statement timeout occurred, please set the variable ob_query_timeout to a larger value an then restart the statement at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:90) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:70) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:79) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:88) ~[mybatis-spring-2.0.4.jar!/:2.0.4] at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:440) ~[mybatis-spring-2.0.4.jar!/:2.0.4] at com.sun.proxy.$Proxy324.selectOne(Unknown Source) ~[na:na] at org.mybatis.spring.SqlSessionTemplate.selectOne(SqlSessionTemplate.java:159) ~[mybatis-spring-2.0.4.jar!/:2.0.4] at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:87) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:152) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:85) ~[mybatis-3.5.6.jar!/:3.5.6] at com.sun.proxy.$Proxy533.getLastExecutionForUpdate(Unknown Source) ~[na:na] at com.oceanbase.ocp.perf.sql.diag.internal.access.OcpPerfSqlDiagnosisTaskDbAccessImpl.getLastExecutionForUpdate(OcpPerfSqlDiagnosisTaskDbAccessImpl.java:40) ~[ocp-perf-sql-diag-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.perf.sql.diag.internal.OcpPerfSqlDiagnosisTaskService.getNextExecution(OcpPerfSqlDiagnosisTaskService.java:40) ~[ocp-perf-sql-diag-pipeline-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.perf.sql.diag.internal.OcpPerfSqlDiagnosisTaskService$$FastClassBySpringCGLIB$$a1a13d9d.invoke() ~[ocp-perf-sql-diag-pipeline-4.0.3-20230301.jar!/:4.0.3-20230301] at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) ~[spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) ~[spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) ~[spring-tx-5.3.23.jar!/:5.3.23] at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) ~[spring-tx-5.3.23.jar!/:5.3.23] at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) ~[spring-tx-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) ~[spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) ~[spring-aop-5.3.23.jar!/:5.3.23] at com.oceanbase.ocp.perf.sql.diag.internal.OcpPerfSqlDiagnosisTaskService$$EnhancerBySpringCGLIB$$e88b9a71.getNextExecution() ~[ocp-perf-sql-diag-pipeline-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.perf.sql.diag.internal.AbstractDiagnoseService.lambda$init$1(AbstractDiagnoseService.java:138) ~[ocp-perf-sql-diag-pipeline-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_312] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[na:1.8.0_312] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_312] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLInvalidAuthorizationSpecException: (conn=3145736) Statement timeout occurred, please set the variable ob_query_timeout to a larger value an then restart the statement at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:114) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:202) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseStatement.executeExceptionEpilogue(OceanBaseStatement.java:315) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:243) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.JDBC4PreparedStatement.execute(JDBC4PreparedStatement.java:161) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidPooledPreparedStatement.execute(DruidPooledPreparedStatement.java:497) ~[druid-1.2.8.jar!/:1.2.8] at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:64) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:79) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:63) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:325) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:89) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:147) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:76) ~[mybatis-3.5.6.jar!/:3.5.6] at sun.reflect.GeneratedMethodAccessor463.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_312] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_312] at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:426) ~[mybatis-spring-2.0.4.jar!/:2.0.4] ... 28 common frames omitted Caused by: com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException: Statement timeout occurred, please set the variable ob_query_timeout to a larger value an then restart the statement at com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException.of(OceanBaseSqlException.java:79) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:199) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:180) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:372) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:234) ~[oceanbase-client-2.4.2.jar!/:na] ... 44 common frames omitted Caused by: java.sql.SQLException: Statement timeout occurred, please set the variable ob_query_timeout to a larger value an then restart the statement at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:2192) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2057) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:1951) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:370) ~[oceanbase-client-2.4.2.jar!/:na] ... 45 common frames omitted 2023-11-30 16:13:29.524 INFO 12 --- [pool-ocp-schedules-6,,1bc89835b031] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[997e1fae-052d-4a24-a32c-2dd49f574f28, a1cfb9f7-f190-4ebe-9f7b-70b8f35ffcf1] 2023-11-30 16:13:29.525 INFO 12 --- [pool-ocp-schedules-6,,1bc89835b031] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[997e1fae-052d-4a24-a32c-2dd49f574f28, a1cfb9f7-f190-4ebe-9f7b-70b8f35ffcf1] 2023-11-30 16:13:30.335 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:13:30.336 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:13:30.616 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:12:31.0, nextRun=2023-11-30 16:13:30.0, currentDate=Thu Nov 30 16:13:30 CST 2023 2023-11-30 16:13:30.617 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-11-30 16:12:31.0, nextRunTime=2023-11-30 16:13:30.0} 2023-11-30 16:13:30.625 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=1 2023-11-30 16:13:30.626 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:13:30.626 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-11-30 16:13:30.648 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:13:00.0, nextRun=2023-11-30 16:13:30.0, currentDate=Thu Nov 30 16:13:30 CST 2023 2023-11-30 16:13:30.649 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-11-30 16:13:00.0, nextRunTime=2023-11-30 16:13:30.0} 2023-11-30 16:13:30.654 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-11-30 16:13:30.654 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:13:30.655 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-11-30 16:13:30.663 INFO 12 --- [alarm-task-1,,89b05ccdc1eb] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-30 16:13:30.693 INFO 12 --- [alarm-task-2,,8ed9b273a5a2] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-30T16:13:30.693223+08:00, lastProcessTime=2023-11-30T16:13:20.689660+08:00 2023-11-30 16:13:30.788 INFO 12 --- [alarm-task-2,,8ed9b273a5a2] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-30T16:13:30.693223+08:00 2023-11-30 16:13:30.817 INFO 12 --- [alarm-detect-12,,bea7be45a355] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1701331990 ~ 1701332000 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={load1_per_cpu > 2=[ExpressionEvaluateResult(matchValue=2.0, metric=load1_per_cpu, match=true, value=2.85, values={1701332000=2.85}, valueCount=1, labels={host=10.38.68.74, host_name=az-rta-prod-database-database-mysql-13, ob_cluster_id=1, ob_cluster_name=local_v4, obzone=zone1, svr_ip=10.38.68.74})]}, matched=true, message=null, level=2) 2023-11-30 16:13:30.817 INFO 12 --- [alarm-detect-12,,bea7be45a355] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1701331990 ~ 1701332000 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=2.85, values={1701332000=2.85}, valueCount=1, labels={host=10.38.68.74, host_name=az-rta-prod-database-database-mysql-13, ob_cluster_id=1, ob_cluster_name=local_v4, obzone=zone1, svr_ip=10.38.68.74})]}, matched=true, message=null, level=3) 2023-11-30 16:13:31.278 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3152, operation:EXECUTE, state: RUNNING 2023-11-30 16:13:31.286 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3152, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-11-30T16:13:31.273+08:00, endTime=null} 2023-11-30 16:13:31.306 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3151, operation:EXECUTE, state: RUNNING 2023-11-30 16:13:31.311 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3151, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-11-30T16:13:31.302+08:00, endTime=null} 2023-11-30 16:13:31.336 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,97319f577710] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3152, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=3105, task_operation=execute, latest_execution_start_time=2023-11-30T16:13:31.273+08:00, sub_task_instance_id=3152}, listMap={}}, executor=10.38.66.53 2023-11-30 16:13:31.337 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,97319f577710] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-11-30 16:13:31.353 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3151, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=3104, task_operation=execute, latest_execution_start_time=2023-11-30T16:13:31.302+08:00, sub_task_instance_id=3151}, listMap={cluster_ids=[1]}}, executor=10.38.66.53 2023-11-30 16:13:31.354 INFO 12 --- [pool-overview-container-7,,8ed47e308f92] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=3, alarmHistory size=6 2023-11-30 16:13:31.363 INFO 12 --- [pool-49059-thread-2,8984c9bcc2be4c9c,d2818a3cb287] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:local_v4, tenant:sys 2023-11-30 16:13:31.363 INFO 12 --- [pool-49059-thread-1,8984c9bcc2be4c9c,e2033293b645] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:local_v4, tenant:ocp 2023-11-30 16:13:31.385 INFO 12 --- [pool-49059-thread-2,8984c9bcc2be4c9c,d2818a3cb287] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:31.388 INFO 12 --- [pool-49059-thread-2,8984c9bcc2be4c9c,d2818a3cb287] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:31.388 INFO 12 --- [pool-49059-thread-2,8984c9bcc2be4c9c,d2818a3cb287] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:31.392 INFO 12 --- [pool-49059-thread-1,8984c9bcc2be4c9c,e2033293b645] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:31.393 INFO 12 --- [pool-49059-thread-2,8984c9bcc2be4c9c,d2818a3cb287] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:13:31.395 INFO 12 --- [pool-49059-thread-1,8984c9bcc2be4c9c,e2033293b645] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:31.395 INFO 12 --- [pool-49059-thread-1,8984c9bcc2be4c9c,e2033293b645] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:31.398 INFO 12 --- [pool-49059-thread-1,8984c9bcc2be4c9c,e2033293b645] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:13:31.400 INFO 12 --- [pool-49059-thread-2,8984c9bcc2be4c9c,d2818a3cb287] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:31.400 INFO 12 --- [pool-49059-thread-2,8984c9bcc2be4c9c,d2818a3cb287] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:31.402 INFO 12 --- [pool-49059-thread-2,8984c9bcc2be4c9c,d2818a3cb287] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-11-30 16:13:31.402 INFO 12 --- [pool-49059-thread-1,8984c9bcc2be4c9c,e2033293b645] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:31.402 INFO 12 --- [pool-49059-thread-1,8984c9bcc2be4c9c,e2033293b645] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:31.404 INFO 12 --- [pool-49059-thread-1,8984c9bcc2be4c9c,e2033293b645] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-11-30 16:13:31.409 INFO 12 --- [pool-49059-thread-1,8984c9bcc2be4c9c,e2033293b645] o.i.t.TenantCompactionCollectServiceImpl : Cluster never major compaction, just skip 2023-11-30 16:13:31.412 INFO 12 --- [pool-49059-thread-2,8984c9bcc2be4c9c,d2818a3cb287] o.i.t.TenantCompactionCollectServiceImpl : Cluster never major compaction, just skip 2023-11-30 16:13:31.414 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,97319f577710] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-11-30 16:13:31.425 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,97319f577710] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3152, operation:EXECUTE, state: SUCCESSFUL 2023-11-30 16:13:31.476 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:31.478 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:31.479 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:31.481 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:13:31.485 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:31.485 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:31.488 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-11-30 16:13:31.495 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:31.496 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:31.498 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-11-30 16:13:31.520 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:31.520 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:31.524 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-11-30 16:13:31.543 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:31.544 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:31.547 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-11-30 16:13:31.570 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:31.570 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:31.573 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-11-30 16:13:31.590 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-11-30 16:13:31.628 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=local_v4, tenantName=sys, dbUser=ocp_monitor 2023-11-30 16:13:31.635 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:31.639 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:31.640 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:31.645 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:13:31.650 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:31.650 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:31.653 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-11-30 16:13:31.685 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=local_v4, tenantName=sys, dbUser=ocp_monitor 2023-11-30 16:13:31.690 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:31.692 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:31.693 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:31.695 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:13:31.701 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:31.701 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:31.703 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-30 16:13:31.743 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=local_v4, tenantName=sys, dbUser=ocp_monitor 2023-11-30 16:13:31.747 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:31.749 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:31.750 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:31.752 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:13:31.759 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:31.759 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:31.762 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-11-30 16:13:31.768 INFO 12 --- [pool-ocp-schedules-11,,6b62ff9b1824] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-30 16:13:31.793 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=local_v4, tenantName=sys, dbUser=ocp_monitor 2023-11-30 16:13:31.803 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:31.805 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:31.805 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:31.807 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:13:31.813 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:31.813 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:31.816 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-30 16:13:31.818 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-11-30 16:13:31.842 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.38.66.53 2023-11-30 16:13:31.882 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=local_v4, tenantName=sys, dbUser=ocp_monitor 2023-11-30 16:13:31.887 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:31.889 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:31.890 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:31.892 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:13:31.902 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:31.903 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:31.906 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-11-30 16:13:31.921 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.38.66.53, userWhitelist=% 2023-11-30 16:13:31.929 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-11-30 16:13:31.936 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:13:31.936 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:13:31.938 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-11-30 16:13:31.953 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.38.66.53, userWhitelist=% 2023-11-30 16:13:31.958 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-11-30 16:13:31.958 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-11-30 16:13:31.958 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=local_v4, obClusterId=1 2023-11-30 16:13:31.966 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,3ce51373d89f] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3151, operation:EXECUTE, state: SUCCESSFUL 2023-11-30 16:13:31.991 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3104, status:SUCCESS 2023-11-30 16:13:31.995 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3104, status:SUCCESS 2023-11-30 16:13:32.026 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3105, status:SUCCESS 2023-11-30 16:13:32.028 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3105, status:SUCCESS 2023-11-30 16:13:32.208 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.38.66.53:8080 2023-11-30 16:13:32.355 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.38.66.53:8080 2023-11-30 16:13:32.381 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.38.66.53:8080 2023-11-30 16:13:32.405 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.38.66.53:8080 2023-11-30 16:13:33.336 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:13:33.338 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:13:35.133 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.38.66.53:8080 2023-11-30 16:13:35.163 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.38.66.53:8080 2023-11-30 16:13:35.232 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.38.66.53:8080 2023-11-30 16:13:35.240 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.38.66.53:8080 2023-11-30 16:13:36.338 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:13:36.342 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:13:37.237 INFO 12 --- [pool-ocp-schedules-3,,701870ed858c] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=3, alarmHistory size=6 2023-11-30 16:13:37.247 INFO 12 --- [pool-ocp-schedules-3,,701870ed858c] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=3, alarmHistory size=6 2023-11-30 16:13:37.286 INFO 12 --- [pool-ocp-schedules-3,,701870ed858c] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=3, alarmHistory size=6 2023-11-30 16:13:37.295 INFO 12 --- [pool-ocp-schedules-3,,701870ed858c] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=3, alarmHistory size=6 2023-11-30 16:13:37.320 INFO 12 --- [pool-ocp-schedules-3,,701870ed858c] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=3, alarmHistory size=6 2023-11-30 16:13:37.327 INFO 12 --- [pool-ocp-schedules-3,,701870ed858c] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=3, alarmHistory size=6 2023-11-30 16:13:38.370 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.38.66.53:8080 2023-11-30 16:13:39.342 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:13:39.344 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:13:40.663 INFO 12 --- [alarm-task-1,,89b05ccdc1eb] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-30 16:13:40.682 INFO 12 --- [alarm-task-2,,8ed9b273a5a2] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-30T16:13:40.681278+08:00, lastProcessTime=2023-11-30T16:13:30.693223+08:00 2023-11-30 16:13:40.745 INFO 12 --- [alarm-detect-6,,ffb5817f784c] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1701332000 ~ 1701332010 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={load1_per_cpu > 2=[ExpressionEvaluateResult(matchValue=2.0, metric=load1_per_cpu, match=true, value=4.132, values={1701332010=4.132}, valueCount=1, labels={host=10.38.68.74, host_name=az-rta-prod-database-database-mysql-13, ob_cluster_id=1, ob_cluster_name=local_v4, obzone=zone1, svr_ip=10.38.68.74})]}, matched=true, message=null, level=2) 2023-11-30 16:13:40.745 INFO 12 --- [alarm-detect-6,,ffb5817f784c] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1701332000 ~ 1701332010 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=4.132, values={1701332010=4.132}, valueCount=1, labels={host=10.38.68.74, host_name=az-rta-prod-database-database-mysql-13, ob_cluster_id=1, ob_cluster_name=local_v4, obzone=zone1, svr_ip=10.38.68.74})]}, matched=true, message=null, level=3) 2023-11-30 16:13:40.761 INFO 12 --- [alarm-task-2,,8ed9b273a5a2] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-30T16:13:40.681278+08:00 2023-11-30 16:13:41.354 INFO 12 --- [pool-overview-container-5,,40243d027745] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=3, alarmHistory size=6 2023-11-30 16:13:41.552 INFO 12 --- [pool-prometheus-cache-monitor1,,663797064ac7] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1368,"minuteWriteQueue":0,"minuteReadCacheSize":7548,"secondWriteCacheSize":342,"seriesIdKeySize":6161,"secondWriteQueue":0,"minuteWriteCacheSize":6473} 2023-11-30 16:13:41.552 INFO 12 --- [pool-prometheus-cache-monitor1,,663797064ac7] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=6, tcp_retrans=13, ob_unit_num=6, go_memstats_mspan_sys_bytes=8, go_memstats_stack_sys_bytes=8, exporter_export_metrics_bytes_total=16, node_filesystem_size_bytes=34, http_server_requests_seconds_count=87, ob_system_event_total_waits=16, monagent_input_sql_duration_seconds_sum=41, observer_fd_count=6, processor_process_metrics_total=13, ob_plan_cache_hit_total=10, node_filesystem_readonly=32, go_memstats_frees_total=8, sql_audit_input_collect_data_jumped_total=6, io_await=33, host_available=4, go_memstats_mallocs_total=8, ob_unit_config_max_memory_bytes=12, io_util=33, ob_system_event_detail_total_timeouts=802, ob_tenant_disk_v4_data_size=10, ob_latch_sleep_total=153, ob_tenant_unit_min_memory=13, ob_memstore_total_bytes=10, process_resident_memory_bytes=8, ob_cache_size_bytes=19, ob_tenant_assigned_cpu_total=10, ob_tenant_unit_max_memory=13, input_collect_metrics_total=18, ob_tenant_server_available_disk_free=11, node_disk_io_time_weighted_seconds_total=20, node_disk_write_time_seconds_total=20, cpu_count=13, output_write_metrics_total=7, ob_cluster_sync=6, process_open_fds=8, ob_system_event_detail_time_waited_us=802, process_max_fds=8, ob_server_resource_cpu_assigned_percent=6, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=25, go_memstats_heap_inuse_bytes=8, go_memstats_mcache_sys_bytes=8, ob_tenant_disk_v4_log_size=10, node_memory_memfree_bytes=12, process_exists=18, go_gc_duration_seconds=13, ob_server_resource_disk_bytes=6, ocp_alarm_detect_duration_ms_sum=82, node_memory_cached_bytes=12, ob_table_num=10, ob_disk_free_bytes=6, node_filesystem_avail_bytes=34, ob_unit_config_max_iops=12, node_network_transmit_errs_total=16, ob_plan_cache_memory_bytes=10, go_memstats_last_gc_time_seconds=8, go_gc_duration_seconds_count=8, monitor_exporter_avaliable=17, node_disk_read_bytes_total=20, clock_diff_milliseconds=4, ob_compaction_is_suspended=10, mysql_output_write_sql_milliseconds_summary_count=13, ob_tenant_unit_log_disk_size=10, go_memstats_lookups_total=8, mysql_output_write_sql_count=13, log_tailer_count=8, host_agent_process_status=8, ob_server_resource_cpu=6, go_memstats_alloc_bytes=8, ob_server_num=9, node_cpu_seconds_total_native=18, ob_session_all_num=10, ob_tenant_resource_min_memory=10, ob_memstore_freeze_times=10, ob_system_event_time_waited=16, ob_tenant_resource_min_cpu=10, ob_cluster_status_check=7, monagent_plugin_execute_seconds_total=29, node_net_bandwidth_bps=17, monagent_oceanbase_db_pool_idle=1, go_memstats_heap_idle_bytes=8, go_memstats_gc_sys_bytes=8, ob_server_resource_memory_bytes=6, ob_latch_get_total=153, go_info=9, ob_tenant_assigned_mem_assigned=10, log_tailer_processed_file_count=6, node_load15=12, ob_tenant_server_required_size=13, ob_unit_config_min_iops=12, ob_tenant_unit_max_cpu=13, ob_latch_wait_seconds_total=153, node_memory_buffers_bytes=12, go_memstats_heap_released_bytes=8, node_network_receive_bytes_total=16, ob_memstore_freeze_trigger_bytes=10, ob_waitevent_wait_seconds_total=10, ob_server_stopped_duration_seconds=7, go_memstats_next_gc_bytes=8, ob_session_active_num=10, node_filesystem_files_free=42, node_disk_read_time_seconds_total=20, ob_compaction_frozen_version=10, ob_tenant_unit_log_disk_in_use=10, process_virtual_memory_bytes=8, ob_tenant_assigned_cpu_assigned=10, ob_tenant_unit_min_cpu=13, ob_unit_config_max_cpu=12, http_request_milliseconds_summary_count=28, monagent_oceanbase_audit_timeinfo_cache_size=1, monagent_input_sql_duration_seconds_bucket=51, ob_latch_immediate_get_total=153, node_cpu_seconds_total=20, http_request_milliseconds_summary=31, ob_latch_immediate_miss_total=153, ob_index_error_num=6, ob_tenant_server_available_disk_total=11, ob_latch_spin_get_total=153, ob_tenant_memory_hold_bytes=6, go_memstats_buck_hash_sys_bytes=8, monagent_pipeline_report_metrics_total=9, mysql_output_write_sql_milliseconds_summary_sum=13, ob_tenant_disk_total_size=8, mysql_output_write_task_size=13, ob_tenant_context_memory_count=235, node_disk_reads_completed_total=20, exporter_export_metrics_total=16, ob_sysstat=108, node_ntp_offset_seconds=13, ob_latch_miss_total=153, go_memstats_heap_sys_bytes=8, ocp_meta_db_health=4, ob_disk_total_bytes=6, go_memstats_heap_objects=8, go_memstats_mcache_inuse_bytes=8, go_memstats_other_sys_bytes=8, go_threads=8, http_request_milliseconds_summary_sum=28, http_server_requests_seconds_sum=87, ob_memstore_active_bytes=10, go_memstats_alloc_bytes_total=8, ob_tenant_assigned_mem_total=10, ob_tenant_server_partition_replica_count=13, node_memory_memtotal_bytes=12, node_network_transmit_drop_total=16, ob_system_event_detail_total_waits=802, ob_waitevent_wait_total=10, ob_tenant500_memory_used_bytes=6, node_network_transmit_bytes_total=16, ob_compaction_is_error=10, node_network_receive_drop_total=16, monagent_pipeline_execute_total=16, node_filesystem_files=42, process_start_time_seconds=8, go_memstats_heap_alloc_bytes=8, ob_tenant_resource_max_cpu=10, go_gc_duration_seconds_sum=8, go_memstats_mspan_inuse_bytes=8, mysql_output_write_sql_milliseconds_summary=16, ob_server_resource_memory_assigned_percent=6, monagent_pipeline_execute_seconds_total=16, go_memstats_sys_bytes=8, node_network_receive_errs_total=16, observer_ulimit_max_fd_count=6, ob_plan_cache_access_total=10, ob_tenant_memory_used_bytes=6, ob_tenant_server_row_count=13, monagent_oceanbase_plan_cache_size=1, ob_tenant_server_data_size=13, go_memstats_gc_cpu_fraction=8, monagent_oceanbase_audit_cache_size=1, ob_tenant_context_memory_used_bytes=235, node_disk_writes_completed_total=20, process_cpu_seconds_total=8, node_load1=12, ob_tenant_resource_max_memory=10, go_memstats_stack_inuse_bytes=8, node_load5=12, monagent_input_sql_duration_seconds_count=41, node_disk_written_bytes_total=20, monagent_oceanbase_ash_cache_size=1, ob_tenant_context_memory_hold_bytes=235, go_goroutines=8, ob_compaction_last_version=10, ob_server_resource_cpu_assigned=6, ob_unit_config_min_cpu=12, process_virtual_memory_max_bytes=8, monagent_plugin_execute_total=29, monagent_oceanbase_plan_timeinfo_cache_size=1, ob_tenant500_memory_hold_bytes=6, ob_compaction_current_timestamp=10, oceanbase_connectivity=7} 2023-11-30 16:13:41.802 INFO 12 --- [pool-ocp-schedules-4,,0539ed5b51a8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-30 16:13:42.247 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.38.66.53:8080 2023-11-30 16:13:42.344 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:13:42.346 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:13:42.381 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.38.66.53:8080 2023-11-30 16:13:42.385 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.38.66.53:8080 2023-11-30 16:13:42.416 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.38.66.53:8080 2023-11-30 16:13:44.332 ERROR 12 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Error in executing PLAN_CHANGE_DETECTOR org.springframework.dao.PermissionDeniedDataAccessException: ### Error querying database. Cause: java.sql.SQLInvalidAuthorizationSpecException: (conn=3146158) Statement timeout occurred, please set the variable ob_query_timeout to a larger value an then restart the statement ### The error may exist in com/oceanbase/ocp/perf/sql/diag/monitordb/dao/OcpPerfSqlDiagnosisTaskDao.java (best guess) ### The error may involve com.oceanbase.ocp.perf.sql.diag.monitordb.dao.OcpPerfSqlDiagnosisTaskDao.getLastExecutionForUpdate-Inline ### The error occurred while setting parameters ### SQL: SELECT task_name AS taskName, begin_interval_time AS beginIntervalTime, end_interval_time AS endIntervalTime, executor, create_time as createTime, update_time as updateTime, status FROM ocp_perf_sql_diagnosis_task WHERE task_name = ? ORDER BY end_interval_time DESC LIMIT 1 FOR UPDATE NOWAIT ### Cause: java.sql.SQLInvalidAuthorizationSpecException: (conn=3146158) Statement timeout occurred, please set the variable ob_query_timeout to a larger value an then restart the statement ; (conn=3146158) Statement timeout occurred, please set the variable ob_query_timeout to a larger value an then restart the statement; nested exception is java.sql.SQLInvalidAuthorizationSpecException: (conn=3146158) Statement timeout occurred, please set the variable ob_query_timeout to a larger value an then restart the statement at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:90) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:70) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:79) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:88) ~[mybatis-spring-2.0.4.jar!/:2.0.4] at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:440) ~[mybatis-spring-2.0.4.jar!/:2.0.4] at com.sun.proxy.$Proxy324.selectOne(Unknown Source) ~[na:na] at org.mybatis.spring.SqlSessionTemplate.selectOne(SqlSessionTemplate.java:159) ~[mybatis-spring-2.0.4.jar!/:2.0.4] at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:87) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:152) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:85) ~[mybatis-3.5.6.jar!/:3.5.6] at com.sun.proxy.$Proxy533.getLastExecutionForUpdate(Unknown Source) ~[na:na] at com.oceanbase.ocp.perf.sql.diag.internal.access.OcpPerfSqlDiagnosisTaskDbAccessImpl.getLastExecutionForUpdate(OcpPerfSqlDiagnosisTaskDbAccessImpl.java:40) ~[ocp-perf-sql-diag-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.perf.sql.diag.internal.OcpPerfSqlDiagnosisTaskService.getNextExecution(OcpPerfSqlDiagnosisTaskService.java:40) ~[ocp-perf-sql-diag-pipeline-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.perf.sql.diag.internal.OcpPerfSqlDiagnosisTaskService$$FastClassBySpringCGLIB$$a1a13d9d.invoke() ~[ocp-perf-sql-diag-pipeline-4.0.3-20230301.jar!/:4.0.3-20230301] at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) ~[spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) ~[spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) ~[spring-tx-5.3.23.jar!/:5.3.23] at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) ~[spring-tx-5.3.23.jar!/:5.3.23] at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) ~[spring-tx-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) ~[spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) ~[spring-aop-5.3.23.jar!/:5.3.23] at com.oceanbase.ocp.perf.sql.diag.internal.OcpPerfSqlDiagnosisTaskService$$EnhancerBySpringCGLIB$$e88b9a71.getNextExecution() ~[ocp-perf-sql-diag-pipeline-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.perf.sql.diag.internal.AbstractDiagnoseService.lambda$init$1(AbstractDiagnoseService.java:138) ~[ocp-perf-sql-diag-pipeline-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_312] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[na:1.8.0_312] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_312] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLInvalidAuthorizationSpecException: (conn=3146158) Statement timeout occurred, please set the variable ob_query_timeout to a larger value an then restart the statement at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:114) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:202) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseStatement.executeExceptionEpilogue(OceanBaseStatement.java:315) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:243) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.JDBC4PreparedStatement.execute(JDBC4PreparedStatement.java:161) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidPooledPreparedStatement.execute(DruidPooledPreparedStatement.java:497) ~[druid-1.2.8.jar!/:1.2.8] at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:64) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:79) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:63) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:325) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:89) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:147) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:76) ~[mybatis-3.5.6.jar!/:3.5.6] at sun.reflect.GeneratedMethodAccessor463.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_312] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_312] at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:426) ~[mybatis-spring-2.0.4.jar!/:2.0.4] ... 28 common frames omitted Caused by: com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException: Statement timeout occurred, please set the variable ob_query_timeout to a larger value an then restart the statement at com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException.of(OceanBaseSqlException.java:79) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:199) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:180) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:372) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:234) ~[oceanbase-client-2.4.2.jar!/:na] ... 44 common frames omitted Caused by: java.sql.SQLException: Statement timeout occurred, please set the variable ob_query_timeout to a larger value an then restart the statement at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:2192) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2057) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:1951) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:370) ~[oceanbase-client-2.4.2.jar!/:na] ... 45 common frames omitted 2023-11-30 16:13:45.101 INFO 12 --- [pool-prometheus-second-schedule-2,,23209319b439] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=1, collectTaskCount=1 2023-11-30 16:13:45.105 INFO 12 --- [pool-prometheus-second-schedule-2,,701d134059dd] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=2, collectTaskCount=2 2023-11-30 16:13:45.137 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.38.66.53:8080 2023-11-30 16:13:45.346 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:13:45.347 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:13:48.347 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:13:48.349 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:13:49.531 INFO 12 --- [pool-ocp-schedules-19,,fb78c49ff8d0] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[997e1fae-052d-4a24-a32c-2dd49f574f28, a1cfb9f7-f190-4ebe-9f7b-70b8f35ffcf1] 2023-11-30 16:13:49.532 INFO 12 --- [pool-ocp-schedules-19,,fb78c49ff8d0] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[997e1fae-052d-4a24-a32c-2dd49f574f28, a1cfb9f7-f190-4ebe-9f7b-70b8f35ffcf1] 2023-11-30 16:13:50.663 INFO 12 --- [alarm-task-1,,89b05ccdc1eb] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-30 16:13:50.705 INFO 12 --- [alarm-task-2,,8ed9b273a5a2] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-30T16:13:50.703697+08:00, lastProcessTime=2023-11-30T16:13:40.681278+08:00 2023-11-30 16:13:50.809 INFO 12 --- [alarm-task-2,,8ed9b273a5a2] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-30T16:13:50.703697+08:00 2023-11-30 16:13:50.823 INFO 12 --- [alarm-detect-13,,e0905b0a328c] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1701332010 ~ 1701332020 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={load1_per_cpu > 2=[ExpressionEvaluateResult(matchValue=2.0, metric=load1_per_cpu, match=true, value=5.322, values={1701332020=5.322}, valueCount=1, labels={host=10.38.68.74, host_name=az-rta-prod-database-database-mysql-13, ob_cluster_id=1, ob_cluster_name=local_v4, obzone=zone1, svr_ip=10.38.68.74})]}, matched=true, message=null, level=2) 2023-11-30 16:13:50.823 INFO 12 --- [alarm-detect-13,,e0905b0a328c] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1701332010 ~ 1701332020 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=5.322, values={1701332020=5.322}, valueCount=1, labels={host=10.38.68.74, host_name=az-rta-prod-database-database-mysql-13, ob_cluster_id=1, ob_cluster_name=local_v4, obzone=zone1, svr_ip=10.38.68.74})]}, matched=true, message=null, level=3) 2023-11-30 16:13:51.352 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:13:51.354 INFO 12 --- [pool-overview-container-1,,bc2c6740db90] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=3, alarmHistory size=6 2023-11-30 16:13:51.357 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:13:51.830 INFO 12 --- [pool-ocp-schedules-5,,d3bdd7bf7504] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-30 16:13:52.251 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.38.66.53:8080 2023-11-30 16:13:52.385 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.38.66.53:8080 2023-11-30 16:13:52.397 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.38.66.53:8080 2023-11-30 16:13:52.420 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.38.66.53:8080 2023-11-30 16:13:53.373 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.38.66.53:8080 2023-11-30 16:13:54.361 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:13:54.368 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:13:55.142 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.38.66.53:8080 2023-11-30 16:13:57.369 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:13:57.374 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:14:00.000 INFO 12 --- [pool-ocp-schedules-9,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-11-30 16:14:00.000 INFO 12 --- [pool-ocp-schedules-9,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-11-30 16:14:00.000 INFO 12 --- [pool-ocp-schedules-11,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-11-30 16:14:00.001 INFO 12 --- [pool-ocp-schedules-11,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1701332040 2023-11-30 16:14:00.034 INFO 12 --- [pool-ocp-schedules-9,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-11-30 16:14:00.063 INFO 12 --- [pool-ocp-schedules-8,,a3c8dd9211b3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-11-30 16:14:00.072 INFO 12 --- [pool-ocp-schedules-8,,a3c8dd9211b3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-11-30 16:14:00.375 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:14:00.376 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:14:00.621 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:12:00.0, nextRun=2023-11-30 16:14:00.0, currentDate=Thu Nov 30 16:14:00 CST 2023 2023-11-30 16:14:00.622 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=14, name=Sync root service list, templateName=SyncRsListSchedule, scheduleRule=0 */2 * * * ?, lastRunTime=2023-11-30 16:12:00.0, nextRunTime=2023-11-30 16:14:00.0} 2023-11-30 16:14:00.639 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync root service list, nodeCount=1, taskCount=1 2023-11-30 16:14:00.639 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:14:00.639 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync root service list, taskDefinitionId=14, state=RUNNING, operation=EXECUTE} 2023-11-30 16:14:00.663 INFO 12 --- [alarm-task-1,,89b05ccdc1eb] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-30 16:14:00.663 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:13:00.0, nextRun=2023-11-30 16:14:00.0, currentDate=Thu Nov 30 16:14:00 CST 2023 2023-11-30 16:14:00.664 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-30 16:13:00.0, nextRunTime=2023-11-30 16:14:00.0} 2023-11-30 16:14:00.677 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-11-30 16:14:00.677 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:14:00.677 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-11-30 16:14:00.685 INFO 12 --- [alarm-task-2,,8ed9b273a5a2] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-30T16:14:00.685046+08:00, lastProcessTime=2023-11-30T16:13:50.703697+08:00 2023-11-30 16:14:00.686 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:13:00.0, nextRun=2023-11-30 16:14:00.0, currentDate=Thu Nov 30 16:14:00 CST 2023 2023-11-30 16:14:00.686 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-30 16:13:00.0, nextRunTime=2023-11-30 16:14:00.0} 2023-11-30 16:14:00.698 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-11-30 16:14:00.698 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:14:00.698 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-11-30 16:14:00.708 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:13:00.0, nextRun=2023-11-30 16:14:00.0, currentDate=Thu Nov 30 16:14:00 CST 2023 2023-11-30 16:14:00.708 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-30 16:13:00.0, nextRunTime=2023-11-30 16:14:00.0} 2023-11-30 16:14:00.723 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-11-30 16:14:00.723 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:14:00.723 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-11-30 16:14:00.733 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:13:31.0, nextRun=2023-11-30 16:14:00.0, currentDate=Thu Nov 30 16:14:00 CST 2023 2023-11-30 16:14:00.734 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-11-30 16:13:31.0, nextRunTime=2023-11-30 16:14:00.0} 2023-11-30 16:14:00.740 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-11-30 16:14:00.740 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:14:00.740 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-11-30 16:14:00.760 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:13:00.0, nextRun=2023-11-30 16:14:00.0, currentDate=Thu Nov 30 16:14:00 CST 2023 2023-11-30 16:14:00.760 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=38, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-30 16:13:00.0, nextRunTime=2023-11-30 16:14:00.0} 2023-11-30 16:14:00.773 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-11-30 16:14:00.773 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:14:00.773 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=38, state=RUNNING, operation=EXECUTE} 2023-11-30 16:14:00.784 INFO 12 --- [alarm-task-2,,8ed9b273a5a2] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-30T16:14:00.685046+08:00 2023-11-30 16:14:00.800 INFO 12 --- [alarm-detect-4,,8d836067cf7d] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1701332020 ~ 1701332030 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={load1_per_cpu > 2=[ExpressionEvaluateResult(matchValue=2.0, metric=load1_per_cpu, match=true, value=6.025, values={1701332030=6.025}, valueCount=1, labels={host=10.38.68.74, host_name=az-rta-prod-database-database-mysql-13, ob_cluster_id=1, ob_cluster_name=local_v4, obzone=zone1, svr_ip=10.38.68.74})]}, matched=true, message=null, level=2) 2023-11-30 16:14:00.801 INFO 12 --- [alarm-detect-4,,8d836067cf7d] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1701332020 ~ 1701332030 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=6.025, values={1701332030=6.025}, valueCount=1, labels={host=10.38.68.74, host_name=az-rta-prod-database-database-mysql-13, ob_cluster_id=1, ob_cluster_name=local_v4, obzone=zone1, svr_ip=10.38.68.74})]}, matched=true, message=null, level=3) 2023-11-30 16:14:01.355 INFO 12 --- [pool-overview-container-4,,67b159a1101f] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=3, alarmHistory size=6 2023-11-30 16:14:01.547 INFO 12 --- [pool-prometheus-minute-schedule-2,,bc97cf25e9e0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1701332100, interval=60 2023-11-30 16:14:01.552 INFO 12 --- [pool-prometheus-minute-schedule-1,,3a165a995aa9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1701332100, interval=60 2023-11-30 16:14:01.566 INFO 12 --- [pool-prometheus-minute-schedule-1,,3a165a995aa9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2023-11-30 16:14:01.566 INFO 12 --- [pool-prometheus-minute-schedule-1,,3a165a995aa9] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1701332041, count=2, elapsed=1699630709525 2023-11-30 16:14:01.567 INFO 12 --- [pool-prometheus-minute-schedule-2,,bc97cf25e9e0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=1, collectTaskCount=1 2023-11-30 16:14:01.567 INFO 12 --- [pool-prometheus-minute-schedule-2,,bc97cf25e9e0] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1701332041, count=1, elapsed=1699630709526 2023-11-30 16:14:01.594 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3153, operation:EXECUTE, state: RUNNING 2023-11-30 16:14:01.598 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3153, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=0, startTime=2023-11-30T16:14:01.589+08:00, endTime=null} 2023-11-30 16:14:01.616 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3154, operation:EXECUTE, state: RUNNING 2023-11-30 16:14:01.620 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3154, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-11-30T16:14:01.612+08:00, endTime=null} 2023-11-30 16:14:01.647 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3155, operation:EXECUTE, state: RUNNING 2023-11-30 16:14:01.651 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3155, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-11-30T16:14:01.642+08:00, endTime=null} 2023-11-30 16:14:01.673 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,d75605d916ce] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3153, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=3106, task_operation=execute, latest_execution_start_time=2023-11-30T16:14:01.589+08:00, sub_task_instance_id=3153}, listMap={cluster_ids=[1]}}, executor=10.38.66.53 2023-11-30 16:14:01.674 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,17c7affbdf5d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3154, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=3107, task_operation=execute, latest_execution_start_time=2023-11-30T16:14:01.612+08:00, sub_task_instance_id=3154}, listMap={}}, executor=10.38.66.53 2023-11-30 16:14:01.684 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3156, operation:EXECUTE, state: RUNNING 2023-11-30 16:14:01.686 INFO 12 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-11-30 16:14:01.689 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3156, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-11-30T16:14:01.678+08:00, endTime=null} 2023-11-30 16:14:01.696 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,17c7affbdf5d] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3154, operation:EXECUTE, state: SUCCESSFUL 2023-11-30 16:14:01.698 INFO 12 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-30 16:14:01.702 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,d75605d916ce] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3153, operation:EXECUTE, state: SUCCESSFUL 2023-11-30 16:14:01.715 INFO 12 --- [pool-subtask-executor29,8984c9bcc2be4c9c,6acdc7f7d37d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3155, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=3108, task_operation=execute, latest_execution_start_time=2023-11-30T16:14:01.641+08:00, sub_task_instance_id=3155}, listMap={}}, executor=10.38.66.53 2023-11-30 16:14:01.736 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3157, operation:EXECUTE, state: RUNNING 2023-11-30 16:14:01.745 INFO 12 --- [pool-subtask-executor29,8984c9bcc2be4c9c,6acdc7f7d37d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster local_v4, trans stat result is empty. 2023-11-30 16:14:01.756 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3157, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-11-30T16:14:01.731+08:00, endTime=null} 2023-11-30 16:14:01.765 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,d1a9b2de05ae] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3156, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=3109, task_operation=execute, latest_execution_start_time=2023-11-30T16:14:01.678+08:00, sub_task_instance_id=3156}, listMap={}}, executor=10.38.66.53 2023-11-30 16:14:01.766 INFO 12 --- [pool-subtask-executor29,8984c9bcc2be4c9c,6acdc7f7d37d] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3155, operation:EXECUTE, state: SUCCESSFUL 2023-11-30 16:14:01.767 INFO 12 --- [ForkJoinPool-11-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.38.68.74:62889/api/v1/log/alarms, params:null 2023-11-30 16:14:01.788 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3158, operation:EXECUTE, state: RUNNING 2023-11-30 16:14:01.794 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3158, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-11-30T16:14:01.785+08:00, endTime=null} 2023-11-30 16:14:01.796 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,d1a9b2de05ae] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster local_v4, sql count is 0. 2023-11-30 16:14:01.796 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,d1a9b2de05ae] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster local_v4, result is empty. 2023-11-30 16:14:01.800 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3106, status:SUCCESS 2023-11-30 16:14:01.808 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3106, status:SUCCESS 2023-11-30 16:14:01.810 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,d1a9b2de05ae] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3156, operation:EXECUTE, state: SUCCESSFUL 2023-11-30 16:14:01.823 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,73a6aa53725e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3157, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=3110, task_operation=execute, latest_execution_start_time=2023-11-30T16:14:01.730+08:00, sub_task_instance_id=3157}, listMap={}}, executor=10.38.66.53 2023-11-30 16:14:01.823 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,73a6aa53725e] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-11-30 16:14:01.840 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3107, status:SUCCESS 2023-11-30 16:14:01.842 INFO 12 --- [pool-subtask-executor29,8984c9bcc2be4c9c,1d10ceeb4367] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3158, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=3111, task_operation=execute, latest_execution_start_time=2023-11-30T16:14:01.784+08:00, sub_task_instance_id=3158}, listMap={}}, executor=10.38.66.53 2023-11-30 16:14:01.842 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3107, status:SUCCESS 2023-11-30 16:14:01.844 INFO 12 --- [pool-49211-thread-1,8984c9bcc2be4c9c,9e215a229f6f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:local_v4, tenant:ocp 2023-11-30 16:14:01.844 INFO 12 --- [pool-49211-thread-2,8984c9bcc2be4c9c,622e6fd7af7c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:local_v4, tenant:sys 2023-11-30 16:14:01.858 INFO 12 --- [pool-ocp-schedules-18,,218cbc463b95] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-30 16:14:01.882 INFO 12 --- [pool-49211-thread-2,8984c9bcc2be4c9c,622e6fd7af7c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:01.892 INFO 12 --- [pool-49211-thread-2,8984c9bcc2be4c9c,622e6fd7af7c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:01.892 INFO 12 --- [pool-49211-thread-2,8984c9bcc2be4c9c,622e6fd7af7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:01.894 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3108, status:SUCCESS 2023-11-30 16:14:01.896 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3108, status:SUCCESS 2023-11-30 16:14:01.897 INFO 12 --- [pool-49211-thread-2,8984c9bcc2be4c9c,622e6fd7af7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:14:01.899 INFO 12 --- [pool-subtask-executor29,8984c9bcc2be4c9c,1d10ceeb4367] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3158, operation:EXECUTE, state: SUCCESSFUL 2023-11-30 16:14:01.907 INFO 12 --- [pool-49211-thread-1,8984c9bcc2be4c9c,9e215a229f6f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:01.922 INFO 12 --- [pool-49211-thread-1,8984c9bcc2be4c9c,9e215a229f6f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:01.922 INFO 12 --- [pool-49211-thread-1,8984c9bcc2be4c9c,9e215a229f6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:01.928 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3109, status:SUCCESS 2023-11-30 16:14:01.931 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3109, status:SUCCESS 2023-11-30 16:14:01.931 INFO 12 --- [pool-49211-thread-2,8984c9bcc2be4c9c,622e6fd7af7c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:01.932 INFO 12 --- [pool-49211-thread-2,8984c9bcc2be4c9c,622e6fd7af7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:01.936 INFO 12 --- [pool-49211-thread-2,8984c9bcc2be4c9c,622e6fd7af7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-11-30 16:14:01.944 INFO 12 --- [pool-49211-thread-2,8984c9bcc2be4c9c,622e6fd7af7c] o.i.t.TenantCompactionCollectServiceImpl : Cluster never major compaction, just skip 2023-11-30 16:14:01.951 INFO 12 --- [pool-49211-thread-1,8984c9bcc2be4c9c,9e215a229f6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:14:01.960 INFO 12 --- [pool-49211-thread-1,8984c9bcc2be4c9c,9e215a229f6f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:01.960 INFO 12 --- [pool-49211-thread-1,8984c9bcc2be4c9c,9e215a229f6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:01.965 INFO 12 --- [pool-49211-thread-1,8984c9bcc2be4c9c,9e215a229f6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-11-30 16:14:01.975 INFO 12 --- [pool-49211-thread-1,8984c9bcc2be4c9c,9e215a229f6f] o.i.t.TenantCompactionCollectServiceImpl : Cluster never major compaction, just skip 2023-11-30 16:14:01.978 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,73a6aa53725e] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-11-30 16:14:01.981 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3111, status:SUCCESS 2023-11-30 16:14:01.984 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3111, status:SUCCESS 2023-11-30 16:14:02.016 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,73a6aa53725e] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3157, operation:EXECUTE, state: SUCCESSFUL 2023-11-30 16:14:02.288 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.38.66.53:8080 2023-11-30 16:14:02.397 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.38.66.53:8080 2023-11-30 16:14:02.402 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.38.66.53:8080 2023-11-30 16:14:02.422 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.38.66.53:8080 2023-11-30 16:14:03.112 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3110, status:SUCCESS 2023-11-30 16:14:03.114 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3110, status:SUCCESS 2023-11-30 16:14:03.377 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:14:03.379 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:14:03.725 INFO 12 --- [pool-ocp-schedules-1,,1c8a91f73261] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=586751257718591488 2023-11-30 16:14:05.136 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:13:06.0, nextRun=2023-11-30 16:14:05.0, currentDate=Thu Nov 30 16:14:05 CST 2023 2023-11-30 16:14:05.136 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-11-30 16:13:06.0, nextRunTime=2023-11-30 16:14:05.0} 2023-11-30 16:14:05.148 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.38.66.53:8080 2023-11-30 16:14:05.154 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-11-30 16:14:05.154 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:14:05.154 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-11-30 16:14:05.166 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.38.66.53:8080 2023-11-30 16:14:05.237 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.38.66.53:8080 2023-11-30 16:14:05.244 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.38.66.53:8080 2023-11-30 16:14:05.611 INFO 12 --- [pool-ocp-schedules-9,,bbbb4b6efb25] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-11-30 16:14:05.616 INFO 12 --- [pool-ocp-schedules-9,,bbbb4b6efb25] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=1, successCount=0 2023-11-30 16:14:05.669 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:05.671 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:05.672 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:05.674 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:14:05.679 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:05.679 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:05.681 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-11-30 16:14:05.755 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=local_v4, tenantName=sys, dbUser=ocp_monitor 2023-11-30 16:14:05.759 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:05.761 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:05.761 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:05.768 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:14:05.776 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:05.776 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:05.779 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-11-30 16:14:05.787 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:05.787 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:05.790 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-11-30 16:14:05.793 INFO 12 --- [ForkJoinPool-2070-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.38.68.74:2892, status=ACTIVE, startServiceTime=1701328610966002, active=true 2023-11-30 16:14:05.793 INFO 12 --- [ForkJoinPool-2070-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.38.68.74:2892, stopTime=0, stopped=false 2023-11-30 16:14:05.793 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-30 16:14:05.818 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-30 16:14:05.819 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.38.68.74, port=2892, sqlPort=2891, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T15:16:51+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.38.68.74, port=2892, sqlPort=2891, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T07:16:50.966002Z, stopTime=1970-01-01T00:00Z} 2023-11-30 16:14:05.843 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.38.68.74, port=2892, sqlPort=2891, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T07:16:50.966002Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-30 16:14:05.869 INFO 12 --- [pool-ocp-schedules-6,,15fc6a552e21] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-11-30 16:14:05.870 INFO 12 --- [pool-ocp-schedules-16,,d516e6028763] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-11-30 16:14:05.870 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=local_v4, tenantName=sys, dbUser=ocp_monitor 2023-11-30 16:14:05.873 INFO 12 --- [pool-ocp-schedules-16,,d516e6028763] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=1, successCount=0 2023-11-30 16:14:05.874 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3159, operation:EXECUTE, state: RUNNING 2023-11-30 16:14:05.875 INFO 12 --- [pool-ocp-schedules-6,,15fc6a552e21] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=1, successCount=0 2023-11-30 16:14:05.875 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:05.878 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:05.878 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:05.878 INFO 12 --- [pool-subtask-coordinator4,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3159, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-11-30T16:14:05.867+08:00, endTime=null} 2023-11-30 16:14:05.881 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:14:05.886 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:05.886 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:05.888 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:14:05.910 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=local_v4, obClusterId=1 2023-11-30 16:14:05.910 INFO 12 --- [ForkJoinPool-2070-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=local_v4, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"local_v4","ObClusterId":1,"ObRegion":"local_v4","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.38.68.74:2892","role":"LEADER","sql_port":2891}],"Type":"PRIMARY","timestamp":1701331607673000}, createTime=2023-11-30T15:22:21+08:00, creator=admin, obVersion=4.2.1.1, tenantCount=2) 2023-11-30 16:14:05.912 INFO 12 --- [ForkJoinPool-2072-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:05.914 INFO 12 --- [ForkJoinPool-2072-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:05.914 INFO 12 --- [ForkJoinPool-2072-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:05.916 INFO 12 --- [ForkJoinPool-2072-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:14:05.923 INFO 12 --- [ForkJoinPool-2072-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:05.923 INFO 12 --- [ForkJoinPool-2072-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:05.924 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,e0e6e7ff73d0] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3159, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=3112, task_operation=execute, latest_execution_start_time=2023-11-30T16:14:05.866+08:00, sub_task_instance_id=3159}, listMap={}}, executor=10.38.66.53 2023-11-30 16:14:05.926 INFO 12 --- [ForkJoinPool-2072-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:14:05.929 INFO 12 --- [ForkJoinPool-2072-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2023-11-30 16:14:05.938 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,e0e6e7ff73d0] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3159, operation:EXECUTE, state: SUCCESSFUL 2023-11-30 16:14:06.228 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3112, status:SUCCESS 2023-11-30 16:14:06.231 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3112, status:SUCCESS 2023-11-30 16:14:06.379 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:14:06.381 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:14:08.391 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.38.66.53:8080 2023-11-30 16:14:09.381 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:14:09.383 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:14:09.550 INFO 12 --- [pool-ocp-schedules-3,,3ac87ae3e160] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[997e1fae-052d-4a24-a32c-2dd49f574f28, a1cfb9f7-f190-4ebe-9f7b-70b8f35ffcf1] 2023-11-30 16:14:09.550 INFO 12 --- [pool-ocp-schedules-3,,3ac87ae3e160] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[997e1fae-052d-4a24-a32c-2dd49f574f28, a1cfb9f7-f190-4ebe-9f7b-70b8f35ffcf1] 2023-11-30 16:14:10.663 INFO 12 --- [alarm-task-1,,89b05ccdc1eb] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-30 16:14:10.689 INFO 12 --- [alarm-task-2,,8ed9b273a5a2] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-30T16:14:10.688164+08:00, lastProcessTime=2023-11-30T16:14:00.685046+08:00 2023-11-30 16:14:10.771 INFO 12 --- [alarm-task-2,,8ed9b273a5a2] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-30T16:14:10.688164+08:00 2023-11-30 16:14:10.895 INFO 12 --- [alarm-detect-11,,2c7d8edf8011] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1701332030 ~ 1701332040 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={load1_per_cpu > 2=[ExpressionEvaluateResult(matchValue=2.0, metric=load1_per_cpu, match=true, value=5.777, values={1701332040=5.777}, valueCount=1, labels={host=10.38.68.74, host_name=az-rta-prod-database-database-mysql-13, ob_cluster_id=1, ob_cluster_name=local_v4, obzone=zone1, svr_ip=10.38.68.74})]}, matched=true, message=null, level=2) 2023-11-30 16:14:10.895 INFO 12 --- [alarm-detect-11,,2c7d8edf8011] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1701332030 ~ 1701332040 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=5.777, values={1701332040=5.777}, valueCount=1, labels={host=10.38.68.74, host_name=az-rta-prod-database-database-mysql-13, ob_cluster_id=1, ob_cluster_name=local_v4, obzone=zone1, svr_ip=10.38.68.74})]}, matched=true, message=null, level=3) 2023-11-30 16:14:11.355 INFO 12 --- [pool-overview-container-8,,248c6db5bfdc] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=3, alarmHistory size=6 2023-11-30 16:14:11.913 INFO 12 --- [pool-ocp-schedules-12,,f47007fb18d9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-30 16:14:12.302 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.38.66.53:8080 2023-11-30 16:14:12.383 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:14:12.386 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:14:12.400 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.38.66.53:8080 2023-11-30 16:14:12.405 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.38.66.53:8080 2023-11-30 16:14:12.426 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.38.66.53:8080 2023-11-30 16:14:13.845 INFO 12 --- [pool-ocp-schedules-1,,0e471a306942] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-11-30 16:14:13.846 INFO 12 --- [pool-Check host status15,,dab20f502a00] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.38.68.74, originalStatus=ONLINE 2023-11-30 16:14:13.853 INFO 12 --- [pool-Check host status15,,dab20f502a00] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-30 16:14:13.906 INFO 12 --- [pool-Check host status15,,dab20f502a00] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.38.68.74:62888/api/v1/time, params:null 2023-11-30 16:14:13.910 INFO 12 --- [pool-Check host status15,,dab20f502a00] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=2, ip=10.38.68.74, timestamp=2023-11-30T16:14:13.926351522+08:00 2023-11-30 16:14:13.911 INFO 12 --- [pool-Check host status15,,dab20f502a00] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-11-30T16:14:13.910+08:00 2023-11-30 16:14:13.917 INFO 12 --- [pool-Check host status15,,dab20f502a00] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-11-30T16:14:13.910+08:00 and status AVAILABLE 2023-11-30 16:14:13.928 INFO 12 --- [pool-Check host status15,,dab20f502a00] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.38.68.74], timeoutMillis=20000 2023-11-30 16:14:14.188 INFO 12 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1701331980, nodeCount=184, affectRows=184 2023-11-30 16:14:14.189 INFO 12 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1701331980, nodeCount=184, validCount=184 2023-11-30 16:14:14.199 INFO 12 --- [pool-metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1701331980, nodeCount=158, affectRows=158 2023-11-30 16:14:14.200 INFO 12 --- [pool-metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1701331980, nodeCount=158, validCount=158 2023-11-30 16:14:15.158 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.38.66.53:8080 2023-11-30 16:14:15.260 INFO 12 --- [http-nio-0.0.0.0-8080-exec-6,d644ab2e0c43417b,ac2ca3f413c3] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.38.66.53, traceId=d644ab2e0c43417b, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-11-30 16:14:15.262 INFO 12 --- [http-nio-0.0.0.0-8080-exec-6,d644ab2e0c43417b,ac2ca3f413c3] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.38.66.53, traceId=d644ab2e0c43417b, duration=3 ms] 2023-11-30 16:14:15.386 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:14:15.388 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:14:18.388 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:14:18.390 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:14:20.000 INFO 12 --- [pool-Check host status15,,dab20f502a00] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1701332059 16 16 , error= 2023-11-30 16:14:20.004 INFO 12 --- [pool-Check host status15,,dab20f502a00] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-11-30T16:14:20.002+08:00 2023-11-30 16:14:20.011 INFO 12 --- [pool-Check host status15,,dab20f502a00] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-11-30T16:14:20.002+08:00 and status AVAILABLE 2023-11-30 16:14:20.016 INFO 12 --- [pool-Check host status15,,dab20f502a00] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=2 2023-11-30 16:14:20.042 INFO 12 --- [pool-Check host status15,,dab20f502a00] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-30 16:14:20.093 INFO 12 --- [pool-Check host status15,,dab20f502a00] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.38.68.74:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-11-30 16:14:20.187 INFO 12 --- [pool-Check host status15,,dab20f502a00] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2023-11-30 16:14:20.205 INFO 12 --- [pool-Check host status15,,dab20f502a00] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2023-11-30 16:14:20.229 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:13:21.0, nextRun=2023-11-30 16:14:20.0, currentDate=Thu Nov 30 16:14:20 CST 2023 2023-11-30 16:14:20.229 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-11-30 16:13:21.0, nextRunTime=2023-11-30 16:14:20.0} 2023-11-30 16:14:20.235 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-11-30 16:14:20.235 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:14:20.236 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-11-30 16:14:20.244 INFO 12 --- [pool-Check host status15,,dab20f502a00] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.38.68.74:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-11-30 16:14:20.327 INFO 12 --- [pool-Check host status15,,dab20f502a00] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2023-11-30 16:14:20.346 INFO 12 --- [pool-Check host status15,,dab20f502a00] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2023-11-30 16:14:20.376 INFO 12 --- [pool-Check host status15,,dab20f502a00] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.38.68.74:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-11-30 16:14:20.479 INFO 12 --- [pool-Check host status15,,dab20f502a00] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2023-11-30 16:14:20.504 INFO 12 --- [pool-Check host status15,,dab20f502a00] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2023-11-30 16:14:20.553 INFO 12 --- [pool-Check host status16,,93555d378e33] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.38.68.74, originalStatus=ONLINE 2023-11-30 16:14:20.569 INFO 12 --- [pool-Check host status16,,93555d378e33] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-30 16:14:20.633 INFO 12 --- [pool-Check host status16,,93555d378e33] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.38.68.74:62888/api/v1/agent/status, params:null 2023-11-30 16:14:20.640 INFO 12 --- [pool-Check host status16,,93555d378e33] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.38.68.74 2023-11-30 16:14:20.640 INFO 12 --- [pool-Check host status16,,93555d378e33] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.38.68.74 2023-11-30 16:14:20.640 INFO 12 --- [pool-Check host status16,,93555d378e33] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.38.68.74 2023-11-30 16:14:20.663 INFO 12 --- [alarm-task-1,,89b05ccdc1eb] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-30 16:14:20.695 INFO 12 --- [alarm-task-2,,8ed9b273a5a2] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-30T16:14:20.694422+08:00, lastProcessTime=2023-11-30T16:14:10.688164+08:00 2023-11-30 16:14:20.775 INFO 12 --- [alarm-task-2,,8ed9b273a5a2] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-30T16:14:20.694422+08:00 2023-11-30 16:14:20.822 INFO 12 --- [alarm-detect-2,,8a4c028bed6e] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1701332040 ~ 1701332050 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={load1_per_cpu > 2=[ExpressionEvaluateResult(matchValue=2.0, metric=load1_per_cpu, match=true, value=5.342, values={1701332050=5.342}, valueCount=1, labels={host=10.38.68.74, host_name=az-rta-prod-database-database-mysql-13, ob_cluster_id=1, ob_cluster_name=local_v4, obzone=zone1, svr_ip=10.38.68.74})]}, matched=true, message=null, level=2) 2023-11-30 16:14:20.822 INFO 12 --- [alarm-detect-2,,8a4c028bed6e] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1701332040 ~ 1701332050 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=5.342, values={1701332050=5.342}, valueCount=1, labels={host=10.38.68.74, host_name=az-rta-prod-database-database-mysql-13, ob_cluster_id=1, ob_cluster_name=local_v4, obzone=zone1, svr_ip=10.38.68.74})]}, matched=true, message=null, level=3) 2023-11-30 16:14:21.035 INFO 12 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3160, operation:EXECUTE, state: RUNNING 2023-11-30 16:14:21.039 INFO 12 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3160, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-11-30T16:14:21.030+08:00, endTime=null} 2023-11-30 16:14:21.084 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,5833ba374395] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3160, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=3113, task_operation=execute, latest_execution_start_time=2023-11-30T16:14:21.030+08:00, sub_task_instance_id=3160}, listMap={}}, executor=10.38.66.53 2023-11-30 16:14:21.086 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,5833ba374395] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=1, successCount=0 2023-11-30 16:14:21.086 INFO 12 --- [ForkJoinPool-2075-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-11-30 16:14:21.099 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,5833ba374395] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3160, operation:EXECUTE, state: SUCCESSFUL 2023-11-30 16:14:21.359 INFO 12 --- [pool-overview-container-1,,199a10d6cfbe] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=3, alarmHistory size=6 2023-11-30 16:14:21.390 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:14:21.392 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:14:21.642 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3113, status:SUCCESS 2023-11-30 16:14:21.645 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3113, status:SUCCESS 2023-11-30 16:14:21.944 INFO 12 --- [pool-ocp-schedules-10,,b367988c2735] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-30 16:14:22.345 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.38.66.53:8080 2023-11-30 16:14:22.403 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.38.66.53:8080 2023-11-30 16:14:22.408 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.38.66.53:8080 2023-11-30 16:14:22.431 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.38.66.53:8080 2023-11-30 16:14:23.395 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.38.66.53:8080 2023-11-30 16:14:24.392 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:14:24.394 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:14:25.168 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.38.66.53:8080 2023-11-30 16:14:25.581 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:13:25.0, nextRun=2023-11-30 16:14:25.0, currentDate=Thu Nov 30 16:14:25 CST 2023 2023-11-30 16:14:25.581 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-11-30 16:13:25.0, nextRunTime=2023-11-30 16:14:25.0} 2023-11-30 16:14:25.594 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-11-30 16:14:25.594 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:14:25.594 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-11-30 16:14:26.105 INFO 12 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3161, operation:EXECUTE, state: RUNNING 2023-11-30 16:14:26.118 INFO 12 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3161, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-11-30T16:14:26.101+08:00, endTime=null} 2023-11-30 16:14:26.174 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,471a2d793d1c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3161, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=3114, task_operation=execute, latest_execution_start_time=2023-11-30T16:14:26.101+08:00, sub_task_instance_id=3161}, listMap={host_ids=[2]}}, executor=10.38.66.53 2023-11-30 16:14:26.214 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,471a2d793d1c] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=2, ip=10.38.68.74] desired exporter count: 6, actual exporter count: 6 2023-11-30 16:14:26.228 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,471a2d793d1c] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3161, operation:EXECUTE, state: SUCCESSFUL 2023-11-30 16:14:26.782 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3114, status:SUCCESS 2023-11-30 16:14:26.784 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3114, status:SUCCESS 2023-11-30 16:14:27.394 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:14:27.395 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:14:29.556 INFO 12 --- [pool-ocp-schedules-3,,6eee47863ef6] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[997e1fae-052d-4a24-a32c-2dd49f574f28, a1cfb9f7-f190-4ebe-9f7b-70b8f35ffcf1] 2023-11-30 16:14:29.556 INFO 12 --- [pool-ocp-schedules-3,,6eee47863ef6] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[997e1fae-052d-4a24-a32c-2dd49f574f28, a1cfb9f7-f190-4ebe-9f7b-70b8f35ffcf1] 2023-11-30 16:14:30.395 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:14:30.405 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:14:30.663 INFO 12 --- [alarm-task-1,,89b05ccdc1eb] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-30 16:14:30.683 INFO 12 --- [alarm-task-2,,8ed9b273a5a2] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-30T16:14:30.680055+08:00, lastProcessTime=2023-11-30T16:14:20.694422+08:00 2023-11-30 16:14:30.767 INFO 12 --- [alarm-task-2,,8ed9b273a5a2] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-30T16:14:30.680055+08:00 2023-11-30 16:14:30.777 INFO 12 --- [alarm-detect-9,,038a22735b70] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1701332050 ~ 1701332060 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={load1_per_cpu > 2=[ExpressionEvaluateResult(matchValue=2.0, metric=load1_per_cpu, match=true, value=5.11, values={1701332060=5.11}, valueCount=1, labels={host=10.38.68.74, host_name=az-rta-prod-database-database-mysql-13, ob_cluster_id=1, ob_cluster_name=local_v4, obzone=zone1, svr_ip=10.38.68.74})]}, matched=true, message=null, level=2) 2023-11-30 16:14:30.777 INFO 12 --- [alarm-detect-9,,038a22735b70] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1701332050 ~ 1701332060 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=5.11, values={1701332060=5.11}, valueCount=1, labels={host=10.38.68.74, host_name=az-rta-prod-database-database-mysql-13, ob_cluster_id=1, ob_cluster_name=local_v4, obzone=zone1, svr_ip=10.38.68.74})]}, matched=true, message=null, level=3) 2023-11-30 16:14:31.019 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:13:31.0, nextRun=2023-11-30 16:14:30.0, currentDate=Thu Nov 30 16:14:31 CST 2023 2023-11-30 16:14:31.019 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-11-30 16:13:31.0, nextRunTime=2023-11-30 16:14:30.0} 2023-11-30 16:14:31.031 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=1 2023-11-30 16:14:31.031 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:14:31.032 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-11-30 16:14:31.067 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:14:01.0, nextRun=2023-11-30 16:14:30.0, currentDate=Thu Nov 30 16:14:31 CST 2023 2023-11-30 16:14:31.068 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-11-30 16:14:01.0, nextRunTime=2023-11-30 16:14:30.0} 2023-11-30 16:14:31.074 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-11-30 16:14:31.074 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:14:31.074 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-11-30 16:14:31.188 INFO 12 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3162, operation:EXECUTE, state: RUNNING 2023-11-30 16:14:31.193 INFO 12 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3162, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-11-30T16:14:31.184+08:00, endTime=null} 2023-11-30 16:14:31.213 INFO 12 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3163, operation:EXECUTE, state: RUNNING 2023-11-30 16:14:31.216 INFO 12 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3163, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-11-30T16:14:31.210+08:00, endTime=null} 2023-11-30 16:14:31.231 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3162, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=3115, task_operation=execute, latest_execution_start_time=2023-11-30T16:14:31.183+08:00, sub_task_instance_id=3162}, listMap={cluster_ids=[1]}}, executor=10.38.66.53 2023-11-30 16:14:31.260 INFO 12 --- [pool-subtask-executor29,8984c9bcc2be4c9c,e198a0add963] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3163, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=3116, task_operation=execute, latest_execution_start_time=2023-11-30T16:14:31.210+08:00, sub_task_instance_id=3163}, listMap={}}, executor=10.38.66.53 2023-11-30 16:14:31.260 INFO 12 --- [pool-subtask-executor29,8984c9bcc2be4c9c,e198a0add963] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-11-30 16:14:31.281 INFO 12 --- [pool-49354-thread-2,8984c9bcc2be4c9c,5341fa638057] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:local_v4, tenant:sys 2023-11-30 16:14:31.281 INFO 12 --- [pool-49354-thread-1,8984c9bcc2be4c9c,172bf75eea65] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:local_v4, tenant:ocp 2023-11-30 16:14:31.317 INFO 12 --- [pool-49354-thread-2,8984c9bcc2be4c9c,5341fa638057] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:31.319 INFO 12 --- [pool-49354-thread-2,8984c9bcc2be4c9c,5341fa638057] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:31.319 INFO 12 --- [pool-49354-thread-2,8984c9bcc2be4c9c,5341fa638057] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:31.321 INFO 12 --- [pool-49354-thread-1,8984c9bcc2be4c9c,172bf75eea65] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:31.321 INFO 12 --- [pool-49354-thread-2,8984c9bcc2be4c9c,5341fa638057] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:14:31.323 INFO 12 --- [pool-49354-thread-1,8984c9bcc2be4c9c,172bf75eea65] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:31.323 INFO 12 --- [pool-49354-thread-1,8984c9bcc2be4c9c,172bf75eea65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:31.325 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:31.326 INFO 12 --- [pool-49354-thread-1,8984c9bcc2be4c9c,172bf75eea65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:14:31.327 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:31.327 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:31.329 INFO 12 --- [pool-49354-thread-2,8984c9bcc2be4c9c,5341fa638057] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:31.329 INFO 12 --- [pool-49354-thread-2,8984c9bcc2be4c9c,5341fa638057] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:31.331 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:14:31.331 INFO 12 --- [pool-49354-thread-2,8984c9bcc2be4c9c,5341fa638057] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-11-30 16:14:31.332 INFO 12 --- [pool-49354-thread-1,8984c9bcc2be4c9c,172bf75eea65] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:31.332 INFO 12 --- [pool-49354-thread-1,8984c9bcc2be4c9c,172bf75eea65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:31.334 INFO 12 --- [pool-49354-thread-1,8984c9bcc2be4c9c,172bf75eea65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-11-30 16:14:31.335 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:31.335 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:31.337 INFO 12 --- [pool-49354-thread-2,8984c9bcc2be4c9c,5341fa638057] o.i.t.TenantCompactionCollectServiceImpl : Cluster never major compaction, just skip 2023-11-30 16:14:31.338 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-11-30 16:14:31.340 INFO 12 --- [pool-49354-thread-1,8984c9bcc2be4c9c,172bf75eea65] o.i.t.TenantCompactionCollectServiceImpl : Cluster never major compaction, just skip 2023-11-30 16:14:31.342 INFO 12 --- [pool-subtask-executor29,8984c9bcc2be4c9c,e198a0add963] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-11-30 16:14:31.344 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:31.345 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:31.347 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-11-30 16:14:31.356 INFO 12 --- [pool-overview-container-7,,89d4532a3456] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=3, alarmHistory size=6 2023-11-30 16:14:31.358 INFO 12 --- [pool-subtask-executor29,8984c9bcc2be4c9c,e198a0add963] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3163, operation:EXECUTE, state: SUCCESSFUL 2023-11-30 16:14:31.369 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:31.369 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:31.372 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-11-30 16:14:31.393 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:31.393 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:31.395 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-11-30 16:14:31.413 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:31.413 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:31.415 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-11-30 16:14:31.431 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-11-30 16:14:31.463 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=local_v4, tenantName=sys, dbUser=ocp_monitor 2023-11-30 16:14:31.474 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:31.476 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:31.476 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:31.478 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:14:31.484 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:31.484 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:31.487 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-11-30 16:14:31.531 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=local_v4, tenantName=sys, dbUser=ocp_monitor 2023-11-30 16:14:31.535 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:31.537 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:31.537 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:31.541 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:14:31.547 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:31.547 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:31.549 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-30 16:14:31.593 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=local_v4, tenantName=sys, dbUser=ocp_monitor 2023-11-30 16:14:31.600 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:31.602 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:31.602 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:31.605 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:14:31.610 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:31.611 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:31.615 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-11-30 16:14:31.638 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=local_v4, tenantName=sys, dbUser=ocp_monitor 2023-11-30 16:14:31.642 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:31.644 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:31.644 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:31.646 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:14:31.651 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:31.651 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:31.653 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-11-30 16:14:31.655 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-11-30 16:14:31.662 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.38.66.53 2023-11-30 16:14:31.705 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=local_v4, tenantName=sys, dbUser=ocp_monitor 2023-11-30 16:14:31.710 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:31.712 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:31.712 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:31.714 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:14:31.724 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:31.724 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:31.727 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-11-30 16:14:31.743 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.38.66.53, userWhitelist=% 2023-11-30 16:14:31.748 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-11-30 16:14:31.750 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:14:31.750 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:14:31.753 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-11-30 16:14:31.768 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.38.66.53, userWhitelist=% 2023-11-30 16:14:31.773 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-11-30 16:14:31.773 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-11-30 16:14:31.773 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=local_v4, obClusterId=1 2023-11-30 16:14:31.782 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,2ad8730cc06b] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3162, operation:EXECUTE, state: SUCCESSFUL 2023-11-30 16:14:31.947 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3115, status:SUCCESS 2023-11-30 16:14:31.949 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3115, status:SUCCESS 2023-11-30 16:14:31.970 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3116, status:SUCCESS 2023-11-30 16:14:31.972 INFO 12 --- [pool-ocp-schedules-7,,12c3b2fd52c7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-30 16:14:31.973 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3116, status:SUCCESS 2023-11-30 16:14:32.348 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.38.66.53:8080 2023-11-30 16:14:32.406 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.38.66.53:8080 2023-11-30 16:14:32.414 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.38.66.53:8080 2023-11-30 16:14:32.434 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.38.66.53:8080 2023-11-30 16:14:33.405 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:14:33.407 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:14:35.175 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.38.66.53:8080 2023-11-30 16:14:35.180 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.38.66.53:8080 2023-11-30 16:14:35.240 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.38.66.53:8080 2023-11-30 16:14:35.247 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.38.66.53:8080 2023-11-30 16:14:36.407 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:14:36.408 INFO 12 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:14:37.415 INFO 12 --- [pool-ocp-schedules-12,,887a1ae7416e] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=3, alarmHistory size=6 2023-11-30 16:14:37.421 INFO 12 --- [pool-ocp-schedules-12,,887a1ae7416e] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=3, alarmHistory size=6 2023-11-30 16:14:37.462 INFO 12 --- [pool-ocp-schedules-12,,887a1ae7416e] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=3, alarmHistory size=6 2023-11-30 16:14:37.466 INFO 12 --- [pool-ocp-schedules-12,,887a1ae7416e] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=3, alarmHistory size=6 2023-11-30 16:14:37.493 INFO 12 --- [pool-ocp-schedules-12,,887a1ae7416e] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=3, alarmHistory size=6 2023-11-30 16:14:37.498 INFO 12 --- [pool-ocp-schedules-12,,887a1ae7416e] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=3, alarmHistory size=6 2023-11-30 16:14:38.399 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.38.66.53:8080 2023-11-30 16:14:39.409 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:14:39.410 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:14:40.663 INFO 12 --- [alarm-task-1,,89b05ccdc1eb] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-30 16:14:40.690 INFO 12 --- [alarm-task-2,,8ed9b273a5a2] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-30T16:14:40.688733+08:00, lastProcessTime=2023-11-30T16:14:30.680055+08:00 2023-11-30 16:14:40.789 INFO 12 --- [alarm-task-2,,8ed9b273a5a2] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-30T16:14:40.688733+08:00 2023-11-30 16:14:40.795 INFO 12 --- [alarm-detect-9,,f4037d0a98f0] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1701332060 ~ 1701332070 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={load1_per_cpu > 2=[ExpressionEvaluateResult(matchValue=2.0, metric=load1_per_cpu, match=true, value=4.497, values={1701332070=4.497}, valueCount=1, labels={host=10.38.68.74, host_name=az-rta-prod-database-database-mysql-13, ob_cluster_id=1, ob_cluster_name=local_v4, obzone=zone1, svr_ip=10.38.68.74})]}, matched=true, message=null, level=2) 2023-11-30 16:14:40.795 INFO 12 --- [alarm-detect-9,,f4037d0a98f0] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1701332060 ~ 1701332070 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=4.497, values={1701332070=4.497}, valueCount=1, labels={host=10.38.68.74, host_name=az-rta-prod-database-database-mysql-13, ob_cluster_id=1, ob_cluster_name=local_v4, obzone=zone1, svr_ip=10.38.68.74})]}, matched=true, message=null, level=3) 2023-11-30 16:14:41.356 INFO 12 --- [pool-overview-container-3,,159e0e79ef23] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=3, alarmHistory size=6 2023-11-30 16:14:41.552 INFO 12 --- [pool-prometheus-cache-monitor1,,280b8cb560a3] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1368,"minuteWriteQueue":0,"minuteReadCacheSize":7548,"secondWriteCacheSize":342,"seriesIdKeySize":6161,"secondWriteQueue":0,"minuteWriteCacheSize":6473} 2023-11-30 16:14:41.553 INFO 12 --- [pool-prometheus-cache-monitor1,,280b8cb560a3] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=6, tcp_retrans=13, ob_unit_num=6, go_memstats_mspan_sys_bytes=8, go_memstats_stack_sys_bytes=8, exporter_export_metrics_bytes_total=16, node_filesystem_size_bytes=34, http_server_requests_seconds_count=87, ob_system_event_total_waits=16, monagent_input_sql_duration_seconds_sum=41, observer_fd_count=6, processor_process_metrics_total=13, ob_plan_cache_hit_total=10, node_filesystem_readonly=32, go_memstats_frees_total=8, sql_audit_input_collect_data_jumped_total=6, io_await=33, host_available=4, go_memstats_mallocs_total=8, ob_unit_config_max_memory_bytes=12, io_util=33, ob_system_event_detail_total_timeouts=802, ob_tenant_disk_v4_data_size=10, ob_latch_sleep_total=153, ob_tenant_unit_min_memory=13, ob_memstore_total_bytes=10, process_resident_memory_bytes=8, ob_cache_size_bytes=19, ob_tenant_assigned_cpu_total=10, ob_tenant_unit_max_memory=13, input_collect_metrics_total=18, ob_tenant_server_available_disk_free=11, node_disk_io_time_weighted_seconds_total=20, node_disk_write_time_seconds_total=20, cpu_count=13, output_write_metrics_total=7, ob_cluster_sync=6, process_open_fds=8, ob_system_event_detail_time_waited_us=802, process_max_fds=8, ob_server_resource_cpu_assigned_percent=6, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=25, go_memstats_heap_inuse_bytes=8, go_memstats_mcache_sys_bytes=8, ob_tenant_disk_v4_log_size=10, node_memory_memfree_bytes=12, process_exists=18, go_gc_duration_seconds=13, ob_server_resource_disk_bytes=6, ocp_alarm_detect_duration_ms_sum=82, node_memory_cached_bytes=12, ob_table_num=10, ob_disk_free_bytes=6, node_filesystem_avail_bytes=34, ob_unit_config_max_iops=12, node_network_transmit_errs_total=16, ob_plan_cache_memory_bytes=10, go_memstats_last_gc_time_seconds=8, go_gc_duration_seconds_count=8, monitor_exporter_avaliable=17, node_disk_read_bytes_total=20, clock_diff_milliseconds=4, ob_compaction_is_suspended=10, mysql_output_write_sql_milliseconds_summary_count=13, ob_tenant_unit_log_disk_size=10, go_memstats_lookups_total=8, mysql_output_write_sql_count=13, log_tailer_count=8, host_agent_process_status=8, ob_server_resource_cpu=6, go_memstats_alloc_bytes=8, ob_server_num=9, node_cpu_seconds_total_native=18, ob_session_all_num=10, ob_tenant_resource_min_memory=10, ob_memstore_freeze_times=10, ob_system_event_time_waited=16, ob_tenant_resource_min_cpu=10, ob_cluster_status_check=7, monagent_plugin_execute_seconds_total=29, node_net_bandwidth_bps=17, monagent_oceanbase_db_pool_idle=1, go_memstats_heap_idle_bytes=8, go_memstats_gc_sys_bytes=8, ob_server_resource_memory_bytes=6, ob_latch_get_total=153, go_info=9, ob_tenant_assigned_mem_assigned=10, log_tailer_processed_file_count=6, node_load15=12, ob_tenant_server_required_size=13, ob_unit_config_min_iops=12, ob_tenant_unit_max_cpu=13, ob_latch_wait_seconds_total=153, node_memory_buffers_bytes=12, go_memstats_heap_released_bytes=8, node_network_receive_bytes_total=16, ob_memstore_freeze_trigger_bytes=10, ob_waitevent_wait_seconds_total=10, ob_server_stopped_duration_seconds=7, go_memstats_next_gc_bytes=8, ob_session_active_num=10, node_filesystem_files_free=42, node_disk_read_time_seconds_total=20, ob_compaction_frozen_version=10, ob_tenant_unit_log_disk_in_use=10, process_virtual_memory_bytes=8, ob_tenant_assigned_cpu_assigned=10, ob_tenant_unit_min_cpu=13, ob_unit_config_max_cpu=12, http_request_milliseconds_summary_count=28, monagent_oceanbase_audit_timeinfo_cache_size=1, monagent_input_sql_duration_seconds_bucket=51, ob_latch_immediate_get_total=153, node_cpu_seconds_total=20, http_request_milliseconds_summary=31, ob_latch_immediate_miss_total=153, ob_index_error_num=6, ob_tenant_server_available_disk_total=11, ob_latch_spin_get_total=153, ob_tenant_memory_hold_bytes=6, go_memstats_buck_hash_sys_bytes=8, monagent_pipeline_report_metrics_total=9, mysql_output_write_sql_milliseconds_summary_sum=13, ob_tenant_disk_total_size=8, mysql_output_write_task_size=13, ob_tenant_context_memory_count=235, node_disk_reads_completed_total=20, exporter_export_metrics_total=16, ob_sysstat=108, node_ntp_offset_seconds=13, ob_latch_miss_total=153, go_memstats_heap_sys_bytes=8, ocp_meta_db_health=4, ob_disk_total_bytes=6, go_memstats_heap_objects=8, go_memstats_mcache_inuse_bytes=8, go_memstats_other_sys_bytes=8, go_threads=8, http_request_milliseconds_summary_sum=28, http_server_requests_seconds_sum=87, ob_memstore_active_bytes=10, go_memstats_alloc_bytes_total=8, ob_tenant_assigned_mem_total=10, ob_tenant_server_partition_replica_count=13, node_memory_memtotal_bytes=12, node_network_transmit_drop_total=16, ob_system_event_detail_total_waits=802, ob_waitevent_wait_total=10, ob_tenant500_memory_used_bytes=6, node_network_transmit_bytes_total=16, ob_compaction_is_error=10, node_network_receive_drop_total=16, monagent_pipeline_execute_total=16, node_filesystem_files=42, process_start_time_seconds=8, go_memstats_heap_alloc_bytes=8, ob_tenant_resource_max_cpu=10, go_gc_duration_seconds_sum=8, go_memstats_mspan_inuse_bytes=8, mysql_output_write_sql_milliseconds_summary=16, ob_server_resource_memory_assigned_percent=6, monagent_pipeline_execute_seconds_total=16, go_memstats_sys_bytes=8, node_network_receive_errs_total=16, observer_ulimit_max_fd_count=6, ob_plan_cache_access_total=10, ob_tenant_memory_used_bytes=6, ob_tenant_server_row_count=13, monagent_oceanbase_plan_cache_size=1, ob_tenant_server_data_size=13, go_memstats_gc_cpu_fraction=8, monagent_oceanbase_audit_cache_size=1, ob_tenant_context_memory_used_bytes=235, node_disk_writes_completed_total=20, process_cpu_seconds_total=8, node_load1=12, ob_tenant_resource_max_memory=10, go_memstats_stack_inuse_bytes=8, node_load5=12, monagent_input_sql_duration_seconds_count=41, node_disk_written_bytes_total=20, monagent_oceanbase_ash_cache_size=1, ob_tenant_context_memory_hold_bytes=235, go_goroutines=8, ob_compaction_last_version=10, ob_server_resource_cpu_assigned=6, ob_unit_config_min_cpu=12, process_virtual_memory_max_bytes=8, monagent_plugin_execute_total=29, monagent_oceanbase_plan_timeinfo_cache_size=1, ob_tenant500_memory_hold_bytes=6, ob_compaction_current_timestamp=10, oceanbase_connectivity=7} 2023-11-30 16:14:42.013 INFO 12 --- [pool-ocp-schedules-12,,16177940cfea] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-30 16:14:42.390 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.38.66.53:8080 2023-11-30 16:14:42.410 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.38.66.53:8080 2023-11-30 16:14:42.411 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:14:42.412 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:14:42.416 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.38.66.53:8080 2023-11-30 16:14:42.437 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.38.66.53:8080 2023-11-30 16:14:45.101 INFO 12 --- [pool-prometheus-second-schedule-2,,1f7f0991ec08] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=1, collectTaskCount=1 2023-11-30 16:14:45.103 INFO 12 --- [pool-prometheus-second-schedule-1,,ab1253d23abc] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=2, collectTaskCount=2 2023-11-30 16:14:45.181 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.38.66.53:8080 2023-11-30 16:14:45.412 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:14:45.414 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:14:48.415 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:14:48.416 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:14:49.562 INFO 12 --- [pool-ocp-schedules-18,,6dd6ad7c10e7] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[997e1fae-052d-4a24-a32c-2dd49f574f28, a1cfb9f7-f190-4ebe-9f7b-70b8f35ffcf1] 2023-11-30 16:14:49.563 INFO 12 --- [pool-ocp-schedules-18,,6dd6ad7c10e7] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[997e1fae-052d-4a24-a32c-2dd49f574f28, a1cfb9f7-f190-4ebe-9f7b-70b8f35ffcf1] 2023-11-30 16:14:50.663 INFO 12 --- [alarm-task-1,,89b05ccdc1eb] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-30 16:14:50.684 INFO 12 --- [alarm-task-2,,8ed9b273a5a2] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-30T16:14:50.683641+08:00, lastProcessTime=2023-11-30T16:14:40.688733+08:00 2023-11-30 16:14:50.755 INFO 12 --- [alarm-detect-7,,1b16927cc6c2] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1701332070 ~ 1701332080 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={load1_per_cpu > 2=[ExpressionEvaluateResult(matchValue=2.0, metric=load1_per_cpu, match=true, value=3.92, values={1701332080=3.92}, valueCount=1, labels={host=10.38.68.74, host_name=az-rta-prod-database-database-mysql-13, ob_cluster_id=1, ob_cluster_name=local_v4, obzone=zone1, svr_ip=10.38.68.74})]}, matched=true, message=null, level=2) 2023-11-30 16:14:50.755 INFO 12 --- [alarm-detect-7,,1b16927cc6c2] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1701332070 ~ 1701332080 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=3.92, values={1701332080=3.92}, valueCount=1, labels={host=10.38.68.74, host_name=az-rta-prod-database-database-mysql-13, ob_cluster_id=1, ob_cluster_name=local_v4, obzone=zone1, svr_ip=10.38.68.74})]}, matched=true, message=null, level=3) 2023-11-30 16:14:50.765 INFO 12 --- [alarm-task-2,,8ed9b273a5a2] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-30T16:14:50.683641+08:00 2023-11-30 16:14:51.356 INFO 12 --- [pool-overview-container-5,,662d0b3f7b63] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=3, alarmHistory size=6 2023-11-30 16:14:51.416 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:14:51.417 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:14:52.037 INFO 12 --- [pool-ocp-schedules-7,,f44e9f182653] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-30 16:14:52.396 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.38.66.53:8080 2023-11-30 16:14:52.440 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.38.66.53:8080 2023-11-30 16:14:52.446 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.38.66.53:8080 2023-11-30 16:14:52.462 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.38.66.53:8080 2023-11-30 16:14:53.404 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.38.66.53:8080 2023-11-30 16:14:54.418 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:14:54.419 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:14:55.186 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.38.66.53:8080 2023-11-30 16:14:57.420 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:14:57.424 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:15:00.000 INFO 12 --- [pool-ocp-schedules-13,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-11-30 16:15:00.001 INFO 12 --- [pool-ocp-schedules-13,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1701332100 2023-11-30 16:15:00.000 INFO 12 --- [pool-ocp-schedules-1,,] c.o.o.s.task.scheduled.OcpSchedules : Refresh OCP properties. 2023-11-30 16:15:00.001 INFO 12 --- [pool-ocp-schedules-1,,] c.o.ocp.core.property.PropertyService : Refresh config properties 2023-11-30 16:15:00.000 INFO 12 --- [pool-ocp-schedules-9,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-11-30 16:15:00.004 INFO 12 --- [pool-ocp-schedules-9,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-11-30 16:15:00.011 INFO 12 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : new: [] 2023-11-30 16:15:00.011 INFO 12 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : init spring 2023-11-30 16:15:00.011 INFO 12 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2023-11-30 16:15:00.011 INFO 12 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : starting 2023-11-30 16:15:00.013 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:10:00.0, nextRun=2023-11-30 16:15:00.0, currentDate=Thu Nov 30 16:15:00 CST 2023 2023-11-30 16:15:00.013 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=20, name=Check all cluster configUrl, templateName=CheckClusterConfigUrlSchedule, scheduleRule=0 */5 * * * ?, lastRunTime=2023-11-30 16:10:00.0, nextRunTime=2023-11-30 16:15:00.0} 2023-11-30 16:15:00.018 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=CheckClusterConfigUrlSchedule, nodeCount=1, taskCount=1 2023-11-30 16:15:00.018 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:15:00.018 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=CheckClusterConfigUrlSchedule, taskDefinitionId=20, state=RUNNING, operation=EXECUTE} 2023-11-30 16:15:00.020 INFO 12 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : new: [] 2023-11-30 16:15:00.020 INFO 12 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : init spring 2023-11-30 16:15:00.020 INFO 12 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2023-11-30 16:15:00.020 INFO 12 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : starting 2023-11-30 16:15:00.037 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:10:00.0, nextRun=2023-11-30 16:15:00.0, currentDate=Thu Nov 30 16:15:00 CST 2023 2023-11-30 16:15:00.037 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=21, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2023-11-30 16:10:00.0, nextRunTime=2023-11-30 16:15:00.0} 2023-11-30 16:15:00.044 INFO 12 --- [pool-ocp-schedules-9,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-11-30 16:15:00.066 INFO 12 --- [pool-ocp-schedules-4,,2183d864d639] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-11-30 16:15:00.078 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2023-11-30 16:15:00.078 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:15:00.078 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=SqlParseTaskSchedule, taskDefinitionId=21, state=RUNNING, operation=EXECUTE} 2023-11-30 16:15:00.085 INFO 12 --- [pool-ocp-schedules-4,,2183d864d639] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-11-30 16:15:00.089 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:14:01.0, nextRun=2023-11-30 16:15:00.0, currentDate=Thu Nov 30 16:15:00 CST 2023 2023-11-30 16:15:00.089 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-30 16:14:01.0, nextRunTime=2023-11-30 16:15:00.0} 2023-11-30 16:15:00.095 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-11-30 16:15:00.095 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:15:00.095 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-11-30 16:15:00.105 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:14:01.0, nextRun=2023-11-30 16:15:00.0, currentDate=Thu Nov 30 16:15:00 CST 2023 2023-11-30 16:15:00.105 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-30 16:14:01.0, nextRunTime=2023-11-30 16:15:00.0} 2023-11-30 16:15:00.115 INFO 12 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2023-11-30 16:15:00.127 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-11-30 16:15:00.127 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:15:00.127 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-11-30 16:15:00.131 INFO 12 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2023-11-30 16:15:00.131 INFO 12 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2023-11-30 16:15:00.131 INFO 12 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : set system property ocp.version to 4.0.3 2023-11-30 16:15:00.131 INFO 12 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : set systemProperties as the first PropertySource 2023-11-30 16:15:00.131 INFO 12 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.0.3 2023-11-30 16:15:00.136 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:14:01.0, nextRun=2023-11-30 16:15:00.0, currentDate=Thu Nov 30 16:15:00 CST 2023 2023-11-30 16:15:00.136 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-30 16:14:01.0, nextRunTime=2023-11-30 16:15:00.0} 2023-11-30 16:15:00.143 WARN 12 --- [pool-ocp-schedules-1,,] ironmentPostProcessorApplicationListener : Your Okta Issuer URL is missing. You can copy your domain from the Okta Developer Console. Follow these instructions to find it: https://bit.ly/finding-okta-domain To fix this add the `okta.oauth2.issuer` property to your application environments. 2023-11-30 16:15:00.143 INFO 12 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2023-11-30 16:15:00.148 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-11-30 16:15:00.148 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:15:00.148 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-11-30 16:15:00.157 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:14:31.0, nextRun=2023-11-30 16:15:00.0, currentDate=Thu Nov 30 16:15:00 CST 2023 2023-11-30 16:15:00.157 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-11-30 16:14:31.0, nextRunTime=2023-11-30 16:15:00.0} 2023-11-30 16:15:00.164 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-11-30 16:15:00.164 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:15:00.164 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-11-30 16:15:00.173 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:10:00.0, nextRun=2023-11-30 16:15:00.0, currentDate=Thu Nov 30 16:15:00 CST 2023 2023-11-30 16:15:00.173 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=35, name=Collect all tenants mini compactions, templateName=CollectTenantMiniCompactionSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2023-11-30 16:10:00.0, nextRunTime=2023-11-30 16:15:00.0} 2023-11-30 16:15:00.186 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants mini compactions, nodeCount=1, taskCount=1 2023-11-30 16:15:00.186 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:15:00.186 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants mini compactions, taskDefinitionId=35, state=RUNNING, operation=EXECUTE} 2023-11-30 16:15:00.199 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:10:00.0, nextRun=2023-11-30 16:15:00.0, currentDate=Thu Nov 30 16:15:00 CST 2023 2023-11-30 16:15:00.199 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=37, name=HighRiskSqlSchedule, templateName=HighRiskSqlSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2023-11-30 16:10:00.0, nextRunTime=2023-11-30 16:15:00.0} 2023-11-30 16:15:00.226 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=HighRiskSqlAuditStatTaskSchedule, nodeCount=1, taskCount=1 2023-11-30 16:15:00.226 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:15:00.226 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=HighRiskSqlAuditStatTaskSchedule, taskDefinitionId=37, state=RUNNING, operation=EXECUTE} 2023-11-30 16:15:00.240 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:14:01.0, nextRun=2023-11-30 16:15:00.0, currentDate=Thu Nov 30 16:15:00 CST 2023 2023-11-30 16:15:00.240 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=38, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-11-30 16:14:01.0, nextRunTime=2023-11-30 16:15:00.0} 2023-11-30 16:15:00.247 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-11-30 16:15:00.247 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:15:00.247 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=38, state=RUNNING, operation=EXECUTE} 2023-11-30 16:15:00.345 INFO 12 --- [pool-ocp-schedules-7,,6d54b3dac4a9] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1368,"minuteWriteQueue":0,"minuteReadCacheSize":7548,"secondWriteCacheSize":586,"seriesIdKeySize":6161,"secondWriteQueue":0,"minuteWriteCacheSize":6473} 2023-11-30 16:15:00.345 INFO 12 --- [pool-ocp-schedules-7,,6d54b3dac4a9] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=6, tcp_retrans=13, ob_unit_num=6, go_memstats_mspan_sys_bytes=8, go_memstats_stack_sys_bytes=8, exporter_export_metrics_bytes_total=16, node_filesystem_size_bytes=34, http_server_requests_seconds_count=87, ob_system_event_total_waits=16, monagent_input_sql_duration_seconds_sum=41, observer_fd_count=6, processor_process_metrics_total=13, ob_plan_cache_hit_total=10, node_filesystem_readonly=32, go_memstats_frees_total=8, sql_audit_input_collect_data_jumped_total=6, io_await=33, host_available=4, go_memstats_mallocs_total=8, ob_unit_config_max_memory_bytes=12, io_util=33, ob_system_event_detail_total_timeouts=802, ob_tenant_disk_v4_data_size=10, ob_latch_sleep_total=153, ob_tenant_unit_min_memory=13, ob_memstore_total_bytes=10, process_resident_memory_bytes=8, ob_cache_size_bytes=19, ob_tenant_assigned_cpu_total=10, ob_tenant_unit_max_memory=13, input_collect_metrics_total=18, ob_tenant_server_available_disk_free=11, node_disk_io_time_weighted_seconds_total=20, node_disk_write_time_seconds_total=20, cpu_count=13, output_write_metrics_total=7, ob_cluster_sync=6, process_open_fds=8, ob_system_event_detail_time_waited_us=802, process_max_fds=8, ob_server_resource_cpu_assigned_percent=6, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=25, go_memstats_heap_inuse_bytes=8, go_memstats_mcache_sys_bytes=8, ob_tenant_disk_v4_log_size=10, node_memory_memfree_bytes=12, process_exists=18, go_gc_duration_seconds=13, ob_server_resource_disk_bytes=6, ocp_alarm_detect_duration_ms_sum=82, node_memory_cached_bytes=12, ob_table_num=10, ob_disk_free_bytes=6, node_filesystem_avail_bytes=34, ob_unit_config_max_iops=12, node_network_transmit_errs_total=16, ob_plan_cache_memory_bytes=10, go_memstats_last_gc_time_seconds=8, go_gc_duration_seconds_count=8, monitor_exporter_avaliable=17, node_disk_read_bytes_total=20, clock_diff_milliseconds=4, ob_compaction_is_suspended=10, mysql_output_write_sql_milliseconds_summary_count=13, ob_tenant_unit_log_disk_size=10, go_memstats_lookups_total=8, mysql_output_write_sql_count=13, log_tailer_count=8, host_agent_process_status=8, ob_server_resource_cpu=6, go_memstats_alloc_bytes=8, ob_server_num=9, node_cpu_seconds_total_native=18, ob_session_all_num=10, ob_tenant_resource_min_memory=10, ob_memstore_freeze_times=10, ob_system_event_time_waited=16, ob_tenant_resource_min_cpu=10, ob_cluster_status_check=7, monagent_plugin_execute_seconds_total=29, node_net_bandwidth_bps=17, monagent_oceanbase_db_pool_idle=1, go_memstats_heap_idle_bytes=8, go_memstats_gc_sys_bytes=8, ob_server_resource_memory_bytes=6, ob_latch_get_total=153, go_info=9, ob_tenant_assigned_mem_assigned=10, log_tailer_processed_file_count=6, node_load15=12, ob_tenant_server_required_size=13, ob_unit_config_min_iops=12, ob_tenant_unit_max_cpu=13, ob_latch_wait_seconds_total=153, node_memory_buffers_bytes=12, go_memstats_heap_released_bytes=8, node_network_receive_bytes_total=16, ob_memstore_freeze_trigger_bytes=10, ob_waitevent_wait_seconds_total=10, ob_server_stopped_duration_seconds=7, go_memstats_next_gc_bytes=8, ob_session_active_num=10, node_filesystem_files_free=42, node_disk_read_time_seconds_total=20, ob_compaction_frozen_version=10, ob_tenant_unit_log_disk_in_use=10, process_virtual_memory_bytes=8, ob_tenant_assigned_cpu_assigned=10, ob_tenant_unit_min_cpu=13, ob_unit_config_max_cpu=12, http_request_milliseconds_summary_count=28, monagent_oceanbase_audit_timeinfo_cache_size=1, monagent_input_sql_duration_seconds_bucket=51, ob_latch_immediate_get_total=153, node_cpu_seconds_total=20, http_request_milliseconds_summary=31, ob_latch_immediate_miss_total=153, ob_index_error_num=6, ob_tenant_server_available_disk_total=11, ob_latch_spin_get_total=153, ob_tenant_memory_hold_bytes=6, go_memstats_buck_hash_sys_bytes=8, monagent_pipeline_report_metrics_total=9, mysql_output_write_sql_milliseconds_summary_sum=13, ob_tenant_disk_total_size=8, mysql_output_write_task_size=13, ob_tenant_context_memory_count=235, node_disk_reads_completed_total=20, exporter_export_metrics_total=16, ob_sysstat=108, node_ntp_offset_seconds=13, ob_latch_miss_total=153, go_memstats_heap_sys_bytes=8, ocp_meta_db_health=4, ob_disk_total_bytes=6, go_memstats_heap_objects=8, go_memstats_mcache_inuse_bytes=8, go_memstats_other_sys_bytes=8, go_threads=8, http_request_milliseconds_summary_sum=28, http_server_requests_seconds_sum=87, ob_memstore_active_bytes=10, go_memstats_alloc_bytes_total=8, ob_tenant_assigned_mem_total=10, ob_tenant_server_partition_replica_count=13, node_memory_memtotal_bytes=12, node_network_transmit_drop_total=16, ob_system_event_detail_total_waits=802, ob_waitevent_wait_total=10, ob_tenant500_memory_used_bytes=6, node_network_transmit_bytes_total=16, ob_compaction_is_error=10, node_network_receive_drop_total=16, monagent_pipeline_execute_total=16, node_filesystem_files=42, process_start_time_seconds=8, go_memstats_heap_alloc_bytes=8, ob_tenant_resource_max_cpu=10, go_gc_duration_seconds_sum=8, go_memstats_mspan_inuse_bytes=8, mysql_output_write_sql_milliseconds_summary=16, ob_server_resource_memory_assigned_percent=6, monagent_pipeline_execute_seconds_total=16, go_memstats_sys_bytes=8, node_network_receive_errs_total=16, observer_ulimit_max_fd_count=6, ob_plan_cache_access_total=10, ob_tenant_memory_used_bytes=6, ob_tenant_server_row_count=13, monagent_oceanbase_plan_cache_size=1, ob_tenant_server_data_size=13, go_memstats_gc_cpu_fraction=8, monagent_oceanbase_audit_cache_size=1, ob_tenant_context_memory_used_bytes=235, node_disk_writes_completed_total=20, process_cpu_seconds_total=8, node_load1=12, ob_tenant_resource_max_memory=10, go_memstats_stack_inuse_bytes=8, node_load5=12, monagent_input_sql_duration_seconds_count=41, node_disk_written_bytes_total=20, monagent_oceanbase_ash_cache_size=1, ob_tenant_context_memory_hold_bytes=235, go_goroutines=8, ob_compaction_last_version=10, ob_server_resource_cpu_assigned=6, ob_unit_config_min_cpu=12, process_virtual_memory_max_bytes=8, monagent_plugin_execute_total=29, monagent_oceanbase_plan_timeinfo_cache_size=1, ob_tenant500_memory_hold_bytes=6, ob_compaction_current_timestamp=10, oceanbase_connectivity=7} 2023-11-30 16:15:00.425 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:15:00.426 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:15:00.552 INFO 12 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3168, operation:EXECUTE, state: RUNNING 2023-11-30 16:15:00.558 INFO 12 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3168, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-11-30T16:15:00.534+08:00, endTime=null} 2023-11-30 16:15:00.587 INFO 12 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3169, operation:EXECUTE, state: RUNNING 2023-11-30 16:15:00.592 INFO 12 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3169, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-11-30T16:15:00.582+08:00, endTime=null} 2023-11-30 16:15:00.606 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,9028fac247d7] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3168, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=3121, task_operation=execute, latest_execution_start_time=2023-11-30T16:15:00.531+08:00, sub_task_instance_id=3168}, listMap={}}, executor=10.38.66.53 2023-11-30 16:15:00.628 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,9028fac247d7] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster local_v4, sql count is 0. 2023-11-30 16:15:00.628 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,9028fac247d7] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster local_v4, result is empty. 2023-11-30 16:15:00.640 INFO 12 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3170, operation:EXECUTE, state: RUNNING 2023-11-30 16:15:00.646 INFO 12 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3170, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask, seriesId=0, startTime=2023-11-30T16:15:00.628+08:00, endTime=null} 2023-11-30 16:15:00.655 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,9028fac247d7] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3168, operation:EXECUTE, state: SUCCESSFUL 2023-11-30 16:15:00.663 INFO 12 --- [alarm-task-1,,89b05ccdc1eb] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-30 16:15:00.675 INFO 12 --- [pool-subtask-executor29,8984c9bcc2be4c9c,b5549cb0f503] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3169, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=3122, task_operation=execute, latest_execution_start_time=2023-11-30T16:15:00.582+08:00, sub_task_instance_id=3169}, listMap={}}, executor=10.38.66.53 2023-11-30 16:15:00.675 INFO 12 --- [pool-subtask-executor29,8984c9bcc2be4c9c,b5549cb0f503] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-11-30 16:15:00.689 INFO 12 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3171, operation:EXECUTE, state: RUNNING 2023-11-30 16:15:00.703 INFO 12 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3171, name=High Risk Sql Audit Stat Task, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.HighRiskSqlAuditStatTask, seriesId=0, startTime=2023-11-30T16:15:00.678+08:00, endTime=null} 2023-11-30 16:15:00.719 INFO 12 --- [alarm-task-2,,8ed9b273a5a2] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-30T16:15:00.715543+08:00, lastProcessTime=2023-11-30T16:14:50.683641+08:00 2023-11-30 16:15:00.738 INFO 12 --- [pool-49461-thread-2,8984c9bcc2be4c9c,6e53757ac19e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:local_v4, tenant:sys 2023-11-30 16:15:00.739 INFO 12 --- [pool-49461-thread-1,8984c9bcc2be4c9c,6902f8147c67] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:local_v4, tenant:ocp 2023-11-30 16:15:00.740 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,08c0e1715c74] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3170, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=3123, task_operation=execute, latest_execution_start_time=2023-11-30T16:15:00.628+08:00, sub_task_instance_id=3170}, listMap={}}, executor=10.38.66.53 2023-11-30 16:15:00.740 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,08c0e1715c74] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2023-11-30 16:15:00.758 INFO 12 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3172, operation:EXECUTE, state: RUNNING 2023-11-30 16:15:00.780 INFO 12 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3172, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-11-30T16:15:00.752+08:00, endTime=null} 2023-11-30 16:15:00.789 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3121, status:SUCCESS 2023-11-30 16:15:00.790 INFO 12 --- [pool-49461-thread-1,8984c9bcc2be4c9c,6902f8147c67] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:15:00.792 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3121, status:SUCCESS 2023-11-30 16:15:00.792 INFO 12 --- [pool-49461-thread-1,8984c9bcc2be4c9c,6902f8147c67] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:15:00.792 INFO 12 --- [pool-49461-thread-1,8984c9bcc2be4c9c,6902f8147c67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:15:00.794 INFO 12 --- [pool-49461-thread-1,8984c9bcc2be4c9c,6902f8147c67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:15:00.807 INFO 12 --- [pool-49461-thread-1,8984c9bcc2be4c9c,6902f8147c67] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:15:00.807 INFO 12 --- [pool-49461-thread-1,8984c9bcc2be4c9c,6902f8147c67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:15:00.812 INFO 12 --- [pool-49461-thread-1,8984c9bcc2be4c9c,6902f8147c67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-11-30 16:15:00.817 INFO 12 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3164, operation:EXECUTE, state: RUNNING 2023-11-30 16:15:00.821 INFO 12 --- [pool-49461-thread-1,8984c9bcc2be4c9c,6902f8147c67] o.i.t.TenantCompactionCollectServiceImpl : Cluster never major compaction, just skip 2023-11-30 16:15:00.828 INFO 12 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3164, name=Check root service , state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CheckClusterRootServersTask, seriesId=0, startTime=2023-11-30T16:15:00.812+08:00, endTime=null} 2023-11-30 16:15:00.831 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,a2a41c224256] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3171, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=3124, task_operation=execute, latest_execution_start_time=2023-11-30T16:15:00.676+08:00, sub_task_instance_id=3171}, listMap={cluster_ids=[[1]], tenant_ids=[[2]]}}, executor=10.38.66.53 2023-11-30 16:15:00.831 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,a2a41c224256] c.o.o.p.s.i.t.HighRiskSqlAuditStatTask : start to diagnose[2] 2023-11-30 16:15:00.847 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,a2a41c224256] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_2currentTimeUs:1701332100831000 2023-11-30 16:15:00.847 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,a2a41c224256] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_2diagnosisStartTimeUs:1701330300831000 2023-11-30 16:15:00.871 INFO 12 --- [pool-subtask-executor30,8984c9bcc2be4c9c,442efcc136d7] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3172, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=3125, task_operation=execute, latest_execution_start_time=2023-11-30T16:15:00.751+08:00, sub_task_instance_id=3172}, listMap={}}, executor=10.38.66.53 2023-11-30 16:15:00.881 INFO 12 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3165, operation:EXECUTE, state: RUNNING 2023-11-30 16:15:00.897 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,a2a41c224256] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_2totalLength:0 2023-11-30 16:15:00.898 INFO 12 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3165, name=Sql Parse Task, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=2023-11-30T16:15:00.874+08:00, endTime=null} 2023-11-30 16:15:00.900 INFO 12 --- [pool-49461-thread-2,8984c9bcc2be4c9c,6e53757ac19e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:15:00.902 INFO 12 --- [pool-49461-thread-2,8984c9bcc2be4c9c,6e53757ac19e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:15:00.902 INFO 12 --- [pool-49461-thread-2,8984c9bcc2be4c9c,6e53757ac19e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:15:00.904 INFO 12 --- [pool-49461-thread-2,8984c9bcc2be4c9c,6e53757ac19e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:15:00.911 INFO 12 --- [pool-49461-thread-2,8984c9bcc2be4c9c,6e53757ac19e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:15:00.911 INFO 12 --- [pool-49461-thread-2,8984c9bcc2be4c9c,6e53757ac19e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:15:00.913 INFO 12 --- [alarm-task-2,,8ed9b273a5a2] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-30T16:15:00.715543+08:00 2023-11-30 16:15:00.915 INFO 12 --- [pool-49461-thread-2,8984c9bcc2be4c9c,6e53757ac19e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-11-30 16:15:00.917 INFO 12 --- [pool-subtask-executor31,8984c9bcc2be4c9c,102899f64710] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3164, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=3117, task_operation=execute, latest_execution_start_time=2023-11-30T16:15:00.810+08:00, sub_task_instance_id=3164}, listMap={}}, executor=10.38.66.53 2023-11-30 16:15:00.921 INFO 12 --- [pool-49461-thread-2,8984c9bcc2be4c9c,6e53757ac19e] o.i.t.TenantCompactionCollectServiceImpl : Cluster never major compaction, just skip 2023-11-30 16:15:00.930 INFO 12 --- [pool-subtask-executor29,8984c9bcc2be4c9c,b5549cb0f503] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-11-30 16:15:00.941 INFO 12 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3166, operation:EXECUTE, state: RUNNING 2023-11-30 16:15:00.946 INFO 12 --- [pool-subtask-executor29,8984c9bcc2be4c9c,b5549cb0f503] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3169, operation:EXECUTE, state: SUCCESSFUL 2023-11-30 16:15:00.946 INFO 12 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3166, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-11-30T16:15:00.934+08:00, endTime=null} 2023-11-30 16:15:00.959 INFO 12 --- [pool-subtask-executor28,8984c9bcc2be4c9c,76501cfa97c0] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3165, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=3118, task_operation=execute, latest_execution_start_time=2023-11-30T16:15:00.873+08:00, sub_task_instance_id=3165}, listMap={cluster_ids=[[1]], tenant_ids=[[2]]}}, executor=10.38.66.53 2023-11-30 16:15:00.959 INFO 12 --- [pool-subtask-executor28,8984c9bcc2be4c9c,76501cfa97c0] c.o.o.p.sql.internal.task.SqlParseTask : Parse sql for [2] 2023-11-30 16:15:00.923 INFO 12 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2023-11-30 16:15:00.973 INFO 12 --- [pool-subtask-executor31,8984c9bcc2be4c9c,102899f64710] c.o.o.s.t.b.c.AbstractAllClusterTask : Check root service executed completed, tryCount=1, successCount=0 2023-11-30 16:15:00.975 INFO 12 --- [alarm-detect-10,,835885261c1d] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1701332080 ~ 1701332090 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={load1_per_cpu > 2=[ExpressionEvaluateResult(matchValue=2.0, metric=load1_per_cpu, match=true, value=3.642, values={1701332090=3.642}, valueCount=1, labels={host=10.38.68.74, host_name=az-rta-prod-database-database-mysql-13, ob_cluster_id=1, ob_cluster_name=local_v4, obzone=zone1, svr_ip=10.38.68.74})]}, matched=true, message=null, level=2) 2023-11-30 16:15:00.975 INFO 12 --- [alarm-detect-10,,835885261c1d] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1701332080 ~ 1701332090 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=3.642, values={1701332090=3.642}, valueCount=1, labels={host=10.38.68.74, host_name=az-rta-prod-database-database-mysql-13, ob_cluster_id=1, ob_cluster_name=local_v4, obzone=zone1, svr_ip=10.38.68.74})]}, matched=true, message=null, level=3) 2023-11-30 16:15:00.923 INFO 12 --- [pool-49478-thread-1,8984c9bcc2be4c9c,29dff039106a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2023-11-30 16:15:00.946 INFO 12 --- [pool-subtask-executor30,8984c9bcc2be4c9c,442efcc136d7] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3172, operation:EXECUTE, state: SUCCESSFUL 2023-11-30 16:15:00.988 INFO 12 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3167, operation:EXECUTE, state: RUNNING 2023-11-30 16:15:01.043 INFO 12 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3167, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-11-30T16:15:00.969+08:00, endTime=null} 2023-11-30 16:15:01.068 INFO 12 --- [pool-subtask-executor31,8984c9bcc2be4c9c,102899f64710] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3164, operation:EXECUTE, state: SUCCESSFUL 2023-11-30 16:15:01.078 INFO 12 --- [pool-subtask-executor28,8984c9bcc2be4c9c,76501cfa97c0] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3165, operation:EXECUTE, state: SUCCESSFUL 2023-11-30 16:15:01.081 INFO 12 --- [pool-subtask-executor29,8984c9bcc2be4c9c,a895a4b5e6b5] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3166, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=3119, task_operation=execute, latest_execution_start_time=2023-11-30T16:15:00.932+08:00, sub_task_instance_id=3166}, listMap={}}, executor=10.38.66.53 2023-11-30 16:15:01.112 INFO 12 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-11-30 16:15:01.116 INFO 12 --- [pool-49478-thread-1,8984c9bcc2be4c9c,29dff039106a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:15:01.118 INFO 12 --- [pool-49478-thread-1,8984c9bcc2be4c9c,29dff039106a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:15:01.118 INFO 12 --- [pool-49478-thread-1,8984c9bcc2be4c9c,29dff039106a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:15:01.121 INFO 12 --- [pool-49478-thread-1,8984c9bcc2be4c9c,29dff039106a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:15:01.133 INFO 12 --- [pool-subtask-executor29,8984c9bcc2be4c9c,a895a4b5e6b5] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3166, operation:EXECUTE, state: SUCCESSFUL 2023-11-30 16:15:01.133 INFO 12 --- [ForkJoinPool-2078-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:15:01.134 INFO 12 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-11-30 16:15:01.135 INFO 12 --- [ForkJoinPool-2078-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:15:01.136 INFO 12 --- [ForkJoinPool-2078-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:15:01.140 INFO 12 --- [pool-49478-thread-1,8984c9bcc2be4c9c,29dff039106a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:15:01.140 INFO 12 --- [pool-49478-thread-1,8984c9bcc2be4c9c,29dff039106a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:15:01.141 INFO 12 --- [ForkJoinPool-2078-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:15:01.143 INFO 12 --- [pool-49478-thread-1,8984c9bcc2be4c9c,29dff039106a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1701331201000000, 1701332100000000] 2023-11-30 16:15:01.150 INFO 12 --- [pool-49478-thread-1,8984c9bcc2be4c9c,29dff039106a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2023-11-30 16:15:01.159 INFO 12 --- [ForkJoinPool-2078-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:15:01.160 INFO 12 --- [ForkJoinPool-2078-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:15:01.160 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,08c0e1715c74] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2023-11-30 16:15:01.163 INFO 12 --- [pool-subtask-executor30,8984c9bcc2be4c9c,3606d50fb376] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3167, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=3120, task_operation=execute, latest_execution_start_time=2023-11-30T16:15:00.969+08:00, sub_task_instance_id=3167}, listMap={}}, executor=10.38.66.53 2023-11-30 16:15:01.169 INFO 12 --- [ForkJoinPool-2078-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2023-11-30 16:15:01.176 INFO 12 --- [pool-subtask-executor32,8984c9bcc2be4c9c,08c0e1715c74] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3170, operation:EXECUTE, state: SUCCESSFUL 2023-11-30 16:15:01.181 INFO 12 --- [http-nio-0.0.0.0-8080-exec-1,ea17b6372e114c40,d359e35c4cda] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=local_v4&ObClusterId=1&version=2, client=10.38.66.53, traceId=ea17b6372e114c40, method=OcpApiRtnObj com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=local_v4,1,false,2,] 2023-11-30 16:15:01.186 INFO 12 --- [http-nio-0.0.0.0-8080-exec-1,ea17b6372e114c40,d359e35c4cda] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.38.66.53, traceId=ea17b6372e114c40, duration=5 ms] 2023-11-30 16:15:01.191 INFO 12 --- [ForkJoinPool-2078-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:15:01.191 INFO 12 --- [ForkJoinPool-2078-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:15:01.192 INFO 12 --- [pool-subtask-executor30,8984c9bcc2be4c9c,3606d50fb376] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster local_v4, trans stat result is empty. 2023-11-30 16:15:01.193 INFO 12 --- [ForkJoinPool-2078-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2023-11-30 16:15:01.206 INFO 12 --- [pool-subtask-executor30,8984c9bcc2be4c9c,3606d50fb376] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3167, operation:EXECUTE, state: SUCCESSFUL 2023-11-30 16:15:01.206 INFO 12 --- [ForkJoinPool-2078-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:15:01.206 INFO 12 --- [ForkJoinPool-2078-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:15:01.210 INFO 12 --- [ForkJoinPool-2078-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.38.68.74, 2892] 2023-11-30 16:15:01.212 INFO 12 --- [ForkJoinPool-11-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.38.68.74:62889/api/v1/log/alarms, params:null 2023-11-30 16:15:01.214 INFO 12 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]:connector is evicted from holder's cache. cause=EXPIRED, key=ConnectorKey(protocol=http, hostAddress=10.38.68.74, httpPort=62889, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=basic, basicAuthConfig=BasicAuthConfig(username=ocp_agent), digestAuthConfig=null, formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), posConfig=null, proxy=null) 2023-11-30 16:15:01.285 INFO 12 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]:connector is evicted from holder's cache. cause=EXPIRED, key=ConnectorKey(protocol=http, hostAddress=10.38.68.74, httpPort=62888, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=basic, basicAuthConfig=BasicAuthConfig(username=ocp_agent), digestAuthConfig=null, formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), posConfig=null, proxy=null) 2023-11-30 16:15:01.359 INFO 12 --- [pool-overview-container-6,,22427dfd3d36] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=3, alarmHistory size=6 2023-11-30 16:15:01.548 INFO 12 --- [pool-prometheus-minute-schedule-1,,4f926e3d0c3d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1701332160, interval=60 2023-11-30 16:15:01.551 INFO 12 --- [pool-prometheus-minute-schedule-2,,445aa7a0f91d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1701332160, interval=60 2023-11-30 16:15:01.572 INFO 12 --- [pool-prometheus-minute-schedule-1,,4f926e3d0c3d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=1, collectTaskCount=1 2023-11-30 16:15:01.572 INFO 12 --- [pool-prometheus-minute-schedule-1,,4f926e3d0c3d] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1701332101, count=1, elapsed=1699630769471 2023-11-30 16:15:01.595 INFO 12 --- [pool-prometheus-minute-schedule-2,,445aa7a0f91d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2023-11-30 16:15:01.595 INFO 12 --- [pool-prometheus-minute-schedule-2,,445aa7a0f91d] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1701332101, count=2, elapsed=1699630769494 2023-11-30 16:15:01.936 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3117, status:SUCCESS 2023-11-30 16:15:01.939 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3117, status:SUCCESS 2023-11-30 16:15:01.980 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3118, status:SUCCESS 2023-11-30 16:15:01.984 WARN 12 --- [pool-ocp-schedules-1,,] ironmentPostProcessorApplicationListener : Your Okta Issuer URL is missing. You can copy your domain from the Okta Developer Console. Follow these instructions to find it: https://bit.ly/finding-okta-domain To fix this add the `okta.oauth2.issuer` property to your application environments. 2023-11-30 16:15:01.985 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3118, status:SUCCESS 2023-11-30 16:15:01.986 INFO 12 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.966050714S 2023-11-30 16:15:01.986 INFO 12 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2023-11-30 16:15:02.016 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3119, status:SUCCESS 2023-11-30 16:15:02.019 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3119, status:SUCCESS 2023-11-30 16:15:02.045 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3120, status:SUCCESS 2023-11-30 16:15:02.048 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3120, status:SUCCESS 2023-11-30 16:15:02.061 INFO 12 --- [pool-ocp-schedules-14,,7524c07128f2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-30 16:15:02.095 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3122, status:SUCCESS 2023-11-30 16:15:02.102 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3122, status:SUCCESS 2023-11-30 16:15:02.127 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3123, status:SUCCESS 2023-11-30 16:15:02.129 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3123, status:SUCCESS 2023-11-30 16:15:02.155 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3125, status:SUCCESS 2023-11-30 16:15:02.155 INFO 12 --- [pool-ocp-schedules-1,,] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2023-11-30 16:15:02.158 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3125, status:SUCCESS 2023-11-30 16:15:02.323 INFO 12 --- [pool-ocp-schedules-1,,] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}, BootstrapPropertySource {name='bootstrapProperties-ocp-composite'}, BootstrapPropertySource {name='bootstrapProperties-application-composite'}, BootstrapPropertySource {name='bootstrapProperties-application-default'}] 2023-11-30 16:15:02.395 INFO 12 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2023-11-30 16:15:02.395 INFO 12 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2023-11-30 16:15:02.395 INFO 12 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : set system property ocp.version to 4.0.3 2023-11-30 16:15:02.395 INFO 12 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : set systemProperties as the first PropertySource 2023-11-30 16:15:02.395 INFO 12 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.0.3 2023-11-30 16:15:02.395 INFO 12 --- [pool-ocp-schedules-1,,] o.s.boot.SpringApplication : The following 2 profiles are active: "composite", "default" 2023-11-30 16:15:02.407 INFO 12 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2023-11-30 16:15:02.440 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.38.66.53:8080 2023-11-30 16:15:02.449 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.38.66.53:8080 2023-11-30 16:15:02.456 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.38.66.53:8080 2023-11-30 16:15:02.464 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.38.66.53:8080 2023-11-30 16:15:02.629 INFO 12 --- [pool-ocp-schedules-1,,] o.s.boot.SpringApplication : Started application in 2.618 seconds (JVM running for 19054.68) 2023-11-30 16:15:02.630 INFO 12 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT2.618695101S 2023-11-30 16:15:02.878 INFO 12 --- [pool-ocp-schedules-1,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] driver class=com.oceanbase.jdbc.Driver 2023-11-30 16:15:02.878 INFO 12 --- [pool-ocp-schedules-1,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] username=meta@meta#meta 2023-11-30 16:15:02.878 INFO 12 --- [pool-ocp-schedules-1,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] jdbc url=jdbc:oceanbase://10.38.66.53:2883/starsky?useUnicode=true&characterEncoding=UTF8&connectTimeout=2000&socketTimeout=30000&rewriteBatchedStatements=true 2023-11-30 16:15:02.878 INFO 12 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.DataSourceInterceptor : dataSource metaDataSource already initialized. skipped. class: com.oceanbase.ocp.server.config.wrapper.MetadbDataSourceWrapper 2023-11-30 16:15:03.210 INFO 12 --- [pool-ocp-schedules-1,,] c.o.ocp.core.property.PropertyService : Refresh config properties done 2023-11-30 16:15:03.427 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:15:03.428 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:15:03.727 INFO 12 --- [pool-ocp-schedules-15,,098aa63270bf] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=586751257718591488 2023-11-30 16:15:05.185 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.38.66.53:8080 2023-11-30 16:15:05.190 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.38.66.53:8080 2023-11-30 16:15:05.247 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.38.66.53:8080 2023-11-30 16:15:05.250 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.38.66.53:8080 2023-11-30 16:15:05.635 INFO 12 --- [pool-ocp-schedules-16,,29c58679fbe7] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-11-30 16:15:05.638 INFO 12 --- [pool-ocp-schedules-16,,29c58679fbe7] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=1, successCount=0 2023-11-30 16:15:05.696 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:15:05.699 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:15:05.699 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:15:05.702 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:15:05.714 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:15:05.714 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:15:05.717 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-11-30 16:15:05.745 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-11-30 16:14:05.0, nextRun=2023-11-30 16:15:05.0, currentDate=Thu Nov 30 16:15:05 CST 2023 2023-11-30 16:15:05.745 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-11-30 16:14:05.0, nextRunTime=2023-11-30 16:15:05.0} 2023-11-30 16:15:05.752 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-11-30 16:15:05.752 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-11-30 16:15:05.752 INFO 12 --- [pool-task-coordinator1,8984c9bcc2be4c9c,aec76daeed47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-11-30 16:15:05.788 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=local_v4, tenantName=sys, dbUser=ocp_monitor 2023-11-30 16:15:05.793 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:15:05.795 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:15:05.795 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:15:05.799 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:15:05.805 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:15:05.805 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:15:05.808 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-11-30 16:15:05.821 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:15:05.821 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:15:05.824 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-11-30 16:15:05.827 INFO 12 --- [ForkJoinPool-2079-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.38.68.74:2892, status=ACTIVE, startServiceTime=1701328610966002, active=true 2023-11-30 16:15:05.827 INFO 12 --- [ForkJoinPool-2079-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.38.68.74:2892, stopTime=0, stopped=false 2023-11-30 16:15:05.827 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-11-30 16:15:05.849 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-30 16:15:05.850 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.38.68.74, port=2892, sqlPort=2891, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T15:16:51+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.38.68.74, port=2892, sqlPort=2891, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T07:16:50.966002Z, stopTime=1970-01-01T00:00Z} 2023-11-30 16:15:05.882 INFO 12 --- [pool-ocp-schedules-10,,2cebd8ef1abb] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-11-30 16:15:05.885 INFO 12 --- [pool-ocp-schedules-10,,2cebd8ef1abb] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=1, successCount=0 2023-11-30 16:15:05.887 INFO 12 --- [pool-ocp-schedules-5,,07a07aa33e4e] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-11-30 16:15:05.890 INFO 12 --- [pool-ocp-schedules-5,,07a07aa33e4e] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=1, successCount=0 2023-11-30 16:15:05.890 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.38.68.74, port=2892, sqlPort=2891, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T07:16:50.966002Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-11-30 16:15:05.910 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=local_v4, tenantName=sys, dbUser=ocp_monitor 2023-11-30 16:15:05.915 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:15:05.917 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:15:05.917 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:15:05.919 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:15:05.923 INFO 12 --- [ForkJoinPool-2080-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:15:05.925 INFO 12 --- [ForkJoinPool-2080-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:15:05.925 INFO 12 --- [ForkJoinPool-2080-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:15:05.927 INFO 12 --- [ForkJoinPool-2080-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:15:05.928 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:15:05.929 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:15:05.931 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:15:05.937 INFO 12 --- [ForkJoinPool-2080-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:15:05.937 INFO 12 --- [ForkJoinPool-2080-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:15:05.940 INFO 12 --- [ForkJoinPool-2080-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:15:05.949 INFO 12 --- [ForkJoinPool-2080-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2023-11-30 16:15:05.949 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=local_v4, obClusterId=1 2023-11-30 16:15:05.949 INFO 12 --- [ForkJoinPool-2079-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=local_v4, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"local_v4","ObClusterId":1,"ObRegion":"local_v4","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.38.68.74:2892","role":"LEADER","sql_port":2891}],"Type":"PRIMARY","timestamp":1701331607673000}, createTime=2023-11-30T15:22:21+08:00, creator=admin, obVersion=4.2.1.1, tenantCount=2) 2023-11-30 16:15:06.146 INFO 12 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3173, operation:EXECUTE, state: RUNNING 2023-11-30 16:15:06.149 INFO 12 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=3173, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-11-30T16:15:06.136+08:00, endTime=null} 2023-11-30 16:15:06.191 INFO 12 --- [pool-subtask-executor30,8984c9bcc2be4c9c,faf116820df1] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3173, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=3126, task_operation=execute, latest_execution_start_time=2023-11-30T16:15:06.135+08:00, sub_task_instance_id=3173}, listMap={}}, executor=10.38.66.53 2023-11-30 16:15:06.206 INFO 12 --- [pool-subtask-executor30,8984c9bcc2be4c9c,faf116820df1] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3173, operation:EXECUTE, state: SUCCESSFUL 2023-11-30 16:15:06.351 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3126, status:SUCCESS 2023-11-30 16:15:06.354 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3126, status:SUCCESS 2023-11-30 16:15:06.428 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:15:06.429 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:15:07.605 INFO 12 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2023-11-30 16:15:07.644 INFO 12 --- [sql-rollup-daemon,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=local_v4, tenantName=sys, dbUser=ocp_monitor 2023-11-30 16:15:07.651 INFO 12 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:15:07.658 INFO 12 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:15:07.659 INFO 12 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:15:07.663 INFO 12 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-11-30 16:15:07.682 INFO 12 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-11-30 16:15:07.682 INFO 12 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:15:07.686 INFO 12 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-11-30 16:15:07.695 INFO 12 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (local_v4:1-1) 2023-11-30 16:15:07.714 INFO 12 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.38.66.53, sql port:2883 2023-11-30 16:15:07.714 INFO 12 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-11-30 16:15:07.715 INFO 12 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-11-30 16:15:07.769 INFO 12 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=local_v4:1, server=1, level=0->1, c=0) - source=2023-11-30T16:12:44, start=2023-11-30T16:10:07, end=2023-11-30T16:12:37, updates=85, granule=120000ms, delay=157000ms 2023-11-30 16:15:07.771 INFO 12 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=local_v4:1, server=1, level=0->1, c=1) - not enough cached interval (source=2023-11-30T16:12:44, target=2023-11-30T16:12:37, granule=120000ms, delay=7000ms) 2023-11-30 16:15:07.771 INFO 12 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=local_v4:1, server=1, level=1->2, c=0) - not enough cached interval (source=2023-11-30T16:12:37, target=2023-11-30T16:01:07, granule=600000ms, delay=690000ms) 2023-11-30 16:15:07.772 INFO 12 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=167ms, period=120000ms 2023-11-30 16:15:08.408 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.38.66.53:8080 2023-11-30 16:15:09.430 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:15:09.431 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:15:09.569 INFO 12 --- [pool-ocp-schedules-10,,5bd4af1c579d] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[997e1fae-052d-4a24-a32c-2dd49f574f28, a1cfb9f7-f190-4ebe-9f7b-70b8f35ffcf1] 2023-11-30 16:15:09.569 INFO 12 --- [pool-ocp-schedules-10,,5bd4af1c579d] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[997e1fae-052d-4a24-a32c-2dd49f574f28, a1cfb9f7-f190-4ebe-9f7b-70b8f35ffcf1] 2023-11-30 16:15:10.663 INFO 12 --- [alarm-task-1,,89b05ccdc1eb] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-11-30 16:15:10.728 INFO 12 --- [alarm-task-2,,8ed9b273a5a2] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-11-30T16:15:10.727612+08:00, lastProcessTime=2023-11-30T16:15:00.715543+08:00 2023-11-30 16:15:10.808 INFO 12 --- [alarm-task-2,,8ed9b273a5a2] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-11-30T16:15:10.727612+08:00 2023-11-30 16:15:10.810 WARN 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,a2a41c224256] c.o.o.c.t.engine.runner.RunnerFactory : Execute task failed, subtask=SubtaskInstanceOverview{id=3171, name=High Risk Sql Audit Stat Task, state=FAILED, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.HighRiskSqlAuditStatTask, seriesId=0, startTime=2023-11-30T16:15:00.678+08:00, endTime=null} org.springframework.dao.PermissionDeniedDataAccessException: ### Error updating database. Cause: java.sql.SQLInvalidAuthorizationSpecException: (conn=2098361) Statement timeout occurred, please set the variable ob_query_timeout to a larger value an then restart the statement ### The error may exist in com/oceanbase/ocp/perf/sql/diag/monitordb/dao/OcpPerfSqlDiagnosisTaskDao.java (best guess) ### The error may involve com.oceanbase.ocp.perf.sql.diag.monitordb.dao.OcpPerfSqlDiagnosisTaskDao.save-Inline ### The error occurred while setting parameters ### SQL: REPLACE INTO ocp_perf_sql_diagnosis_task (task_name, begin_interval_time, end_interval_time, executor, status) VALUES (?, ?, ?, ?, ?) ### Cause: java.sql.SQLInvalidAuthorizationSpecException: (conn=2098361) Statement timeout occurred, please set the variable ob_query_timeout to a larger value an then restart the statement ; (conn=2098361) Statement timeout occurred, please set the variable ob_query_timeout to a larger value an then restart the statement; nested exception is java.sql.SQLInvalidAuthorizationSpecException: (conn=2098361) Statement timeout occurred, please set the variable ob_query_timeout to a larger value an then restart the statement at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:90) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:70) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:79) ~[spring-jdbc-5.3.23.jar!/:5.3.23] at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:88) ~[mybatis-spring-2.0.4.jar!/:2.0.4] at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:440) ~[mybatis-spring-2.0.4.jar!/:2.0.4] at com.sun.proxy.$Proxy324.insert(Unknown Source) ~[na:na] at org.mybatis.spring.SqlSessionTemplate.insert(SqlSessionTemplate.java:271) ~[mybatis-spring-2.0.4.jar!/:2.0.4] at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:62) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:152) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:85) ~[mybatis-3.5.6.jar!/:3.5.6] at com.sun.proxy.$Proxy533.save(Unknown Source) ~[na:na] at com.oceanbase.ocp.perf.sql.internal.HighRiskSqlInspectService.saveTaskExecuteState(HighRiskSqlInspectService.java:163) ~[ocp-perf-sql-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.perf.sql.internal.HighRiskSqlInspectService.inspect(HighRiskSqlInspectService.java:132) ~[ocp-perf-sql-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.perf.sql.internal.task.HighRiskSqlAuditStatTask.run(HighRiskSqlAuditStatTask.java:41) ~[ocp-perf-sql-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:60) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:111) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.redirectOutputIfNotSysSchedule(RunnerFactory.java:174) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.run(RunnerFactory.java:101) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.ReadySubtaskWorker.lambda$null$1(ReadySubtaskWorker.java:127) ~[ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLInvalidAuthorizationSpecException: (conn=2098361) Statement timeout occurred, please set the variable ob_query_timeout to a larger value an then restart the statement at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:114) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:202) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseStatement.executeExceptionEpilogue(OceanBaseStatement.java:315) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:243) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.JDBC4PreparedStatement.execute(JDBC4PreparedStatement.java:161) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidPooledPreparedStatement.execute(DruidPooledPreparedStatement.java:497) ~[druid-1.2.8.jar!/:1.2.8] at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:47) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:74) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:50) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:117) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:76) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:197) ~[mybatis-3.5.6.jar!/:3.5.6] at org.apache.ibatis.session.defaults.DefaultSqlSession.insert(DefaultSqlSession.java:184) ~[mybatis-3.5.6.jar!/:3.5.6] at sun.reflect.GeneratedMethodAccessor894.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_312] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_312] at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:426) ~[mybatis-spring-2.0.4.jar!/:2.0.4] ... 20 common frames omitted Caused by: com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException: Statement timeout occurred, please set the variable ob_query_timeout to a larger value an then restart the statement at com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException.of(OceanBaseSqlException.java:79) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:199) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:180) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:372) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:234) ~[oceanbase-client-2.4.2.jar!/:na] ... 33 common frames omitted Caused by: java.sql.SQLException: Statement timeout occurred, please set the variable ob_query_timeout to a larger value an then restart the statement at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:2192) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2057) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:1951) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:370) ~[oceanbase-client-2.4.2.jar!/:na] ... 34 common frames omitted 2023-11-30 16:15:10.820 INFO 12 --- [pool-subtask-executor27,8984c9bcc2be4c9c,a2a41c224256] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 3171, operation:EXECUTE, state: FAILED 2023-11-30 16:15:11.032 INFO 12 --- [alarm-detect-6,,f132adce2546] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1701332090 ~ 1701332100 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={load1_per_cpu > 2=[ExpressionEvaluateResult(matchValue=2.0, metric=load1_per_cpu, match=true, value=3.16, values={1701332100=3.16}, valueCount=1, labels={host=10.38.68.74, host_name=az-rta-prod-database-database-mysql-13, ob_cluster_id=1, ob_cluster_name=local_v4, obzone=zone1, svr_ip=10.38.68.74})]}, matched=true, message=null, level=2) 2023-11-30 16:15:11.032 INFO 12 --- [alarm-detect-6,,f132adce2546] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1701332090 ~ 1701332100 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=3.16, values={1701332100=3.16}, valueCount=1, labels={host=10.38.68.74, host_name=az-rta-prod-database-database-mysql-13, ob_cluster_id=1, ob_cluster_name=local_v4, obzone=zone1, svr_ip=10.38.68.74})]}, matched=true, message=null, level=3) 2023-11-30 16:15:11.360 INFO 12 --- [pool-overview-container-8,,fad7c6c0529e] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=3, alarmHistory size=6 2023-11-30 16:15:11.504 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.o.c.t.e.c.worker.RunningTaskWorker : Mark failed task, entity=TaskInstanceEntity{id=3124, name=HighRiskSqlAuditStatTaskSchedule, taskDefinitionId=37, state=RUNNING, operation=EXECUTE} 2023-11-30 16:15:11.513 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3124, status:FAILED 2023-11-30 16:15:11.516 INFO 12 --- [pool-task-coordinator2,faa4463d998e41f8,431d852e9f60] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3124, status:FAILED 2023-11-30 16:15:11.667 INFO 12 --- [http-nio-0.0.0.0-8080-exec-5,69dae5b77f814a1d,41135253296b] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.38.66.53, traceId=69dae5b77f814a1d, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-11-30 16:15:11.670 INFO 12 --- [http-nio-0.0.0.0-8080-exec-5,69dae5b77f814a1d,41135253296b] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.38.66.53, traceId=69dae5b77f814a1d, duration=4 ms] 2023-11-30 16:15:12.121 INFO 12 --- [pool-ocp-schedules-11,,1ed00fa96d36] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-11-30 16:15:12.431 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:15:12.433 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-11-30 16:15:12.449 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.38.66.53:8080 2023-11-30 16:15:12.483 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.38.66.53:8080 2023-11-30 16:15:12.489 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.38.66.53:8080 2023-11-30 16:15:12.491 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.38.66.53:8080 2023-11-30 16:15:14.221 INFO 12 --- [pool-metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1701332040, nodeCount=342, affectRows=342 2023-11-30 16:15:14.223 INFO 12 --- [pool-metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1701332040, nodeCount=342, validCount=342 2023-11-30 16:15:15.199 INFO 12 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.38.66.53:8080 2023-11-30 16:15:15.433 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.38.66.53:8080 refreshes policies... 2023-11-30 16:15:15.436 INFO 12 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0