ERROR 3374086 --- [ocp-schedules-30,a1d22ec2ef647fc3,fcdf5c981973abec] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.config.Task$OutcomeTrackingRunnable.run(Task.java:85) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.lang.reflect.InvocationTargetException: null at jdk.internal.reflect.GeneratedMethodAccessor981.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'com.oceanbase.ocp.selfcure.contingency.ContingencyInternalService' available at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:394) at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:385) at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1290) at com.oceanbase.ocp.core.util.BeanUtils.getBean(BeanUtils.java:33) at com.oceanbase.ocp.service.task.util.Beans.contingencyInternalService(Beans.java:400) at com.oceanbase.ocp.service.task.business.tenant.CheckContingencyTenantSqlPortListenTask.getContingencyParticipantTargetMatcher(CheckContingencyTenantSqlPortListenTask.java:214) at com.oceanbase.ocp.service.task.business.tenant.CheckContingencyTenantSqlPortListenTask.getContingencyParticipantCluster(CheckContingencyTenantSqlPortListenTask.java:188) at com.oceanbase.ocp.service.task.business.tenant.CheckContingencyTenantSqlPortListenTask.getAllShardingClusters(CheckContingencyTenantSqlPortListenTask.java:65) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:63) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.checkContingencyTenantSqlPortListen(TenantSchedules.java:96) ... 14 common frames omitted 2026-01-27 14:28:56.453 INFO 3374086 --- [host-check-host-status1,6a83561149b46a63,7b4839df70e1f79a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2026-01-27 14:28:56.566 INFO 3374086 --- [ocp-schedules-20,14ae8427f3951b4b,f1d6100d6f8379cb] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2026-01-27 14:28:56.567 INFO 3374086 --- [ocp-schedules-20,14ae8427f3951b4b,f1d6100d6f8379cb] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2026-01-27 14:28:57.040 INFO 3374086 --- [ocp-async-6,0b62a2a72b4f81bf,8447bd90d87fa37d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2026-01-27 14:28:58.240 WARN 3374086 --- [ocp-async-7,dbe66d784f0d04aa,dc78a833e8b2821e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:28:58.240 INFO 3374086 --- [ocp-async-10,2ec3e42a14eb9112,9289c0f56ed70012] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:28:58.267 INFO 3374086 --- [ocp-schedules-25,83e571c0a9d721b0,ba0ff441970f2ea3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2026-01-27 14:28:58.308 INFO 3374086 --- [prometheus-minute-schedule-1,d604e816e6fcdf9c,eabb3c71dbd72143] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495340, interval=60 2026-01-27 14:28:58.310 INFO 3374086 --- [prometheus-minute-schedule-1,d604e816e6fcdf9c,eabb3c71dbd72143] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:28:58.560 INFO 3374086 --- [ocp-schedules-21,1f6e5c1c00dc3f11,928c319ed58c2e98] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2026-01-27 14:28:58.560 INFO 3374086 --- [ocp-schedules-21,1f6e5c1c00dc3f11,928c319ed58c2e98] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=872411846587924480 2026-01-27 14:28:58.561 INFO 3374086 --- [ocp-schedules-21,1f6e5c1c00dc3f11,928c319ed58c2e98] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2026-01-27 14:28:58.561 INFO 3374086 --- [ocp-schedules-21,1f6e5c1c00dc3f11,928c319ed58c2e98] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=872606229282988032 2026-01-27 14:28:58.561 INFO 3374086 --- [ocp-schedules-21,1f6e5c1c00dc3f11,928c319ed58c2e98] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2026-01-27 14:28:58.561 INFO 3374086 --- [ocp-schedules-21,1f6e5c1c00dc3f11,928c319ed58c2e98] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=872646545121779712 2026-01-27 14:28:59.024 INFO 3374086 --- [ocp-schedules-7,9309a52af53fdcc7,9e8d50dd386534c3] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2026-01-27 14:28:59.024 INFO 3374086 --- [ocp-schedules-16,d3ec4b43f349f9a7,866c06ebfca6ae94] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2026-01-27 14:28:59.024 INFO 3374086 --- [ocp-schedules-18,f907994474fbf25e,4dcd402c89f818c4] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2026-01-27 14:28:59.324 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:29:00.102 INFO 3374086 --- [prometheus-second-schedule-1,f6dde7e8d9aed273,9147e168f513cd0f] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=1, collectTaskCount=1 2026-01-27 14:29:00.240 INFO 3374086 --- [ocp-async-8,670b514e36829d61,952835cb12a2884c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:29:00.240 WARN 3374086 --- [ocp-async-3,b0e58867191e0315,97884e92ea7b5591] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:29:00.257 INFO 3374086 --- [ocp-schedules-3,25d92259ae569b81,bedb1a678bfb1c30] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2026-01-27 14:29:00.290 INFO 3374086 --- [ocp-schedules-18,4c357e1dd7a855ab,b048a12dfc7c498b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2026-01-27 14:29:00.362 INFO 3374086 --- [ocp-schedules-15,1c6ff7c14cd77f01,eafcdd86370114dd] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2026-01-27 14:29:00.362 INFO 3374086 --- [ocp-schedules-15,1c6ff7c14cd77f01,eafcdd86370114dd] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2026-01-27 14:29:00.363 INFO 3374086 --- [ocp-schedules-27,442d301abec9afb9,e9458fbd42012566] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant tde info status 2026-01-27 14:29:00.363 INFO 3374086 --- [ocp-schedules-27,442d301abec9afb9,e9458fbd42012566] .t.b.t.CheckAllTenantTdeInfoShardingTask : Begin to check all tenant TDE info status 2026-01-27 14:29:00.368 INFO 3374086 --- [ocp-schedules-27,442d301abec9afb9,e9458fbd42012566] .t.b.t.CheckAllTenantTdeInfoShardingTask : Check TDE info for 0 primary tenants with supported OB version 2026-01-27 14:29:00.368 INFO 3374086 --- [ocp-schedules-27,442d301abec9afb9,e9458fbd42012566] .t.b.t.CheckAllTenantTdeInfoShardingTask : Finish to check all tenant TDE info status 2026-01-27 14:29:01.308 INFO 3374086 --- [prometheus-minute-schedule-1,406575d04e4d91a6,a3789c4c0dc59688] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495400, interval=60 2026-01-27 14:29:01.311 INFO 3374086 --- [prometheus-minute-schedule-1,406575d04e4d91a6,a3789c4c0dc59688] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:29:01.882 INFO 3374086 --- [ocp-schedules-27,ad22ea153ba0f58a,84decbc689852a9a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2026-01-27 14:29:01.887 INFO 3374086 --- [ocp-schedules-27,ad22ea153ba0f58a,84decbc689852a9a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2026-01-27 14:29:01.887 INFO 3374086 --- [ocp-schedules-27,ad22ea153ba0f58a,84decbc689852a9a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2026-01-27 14:29:02.240 WARN 3374086 --- [ocp-async-5,cd2a08f038f38851,520893acd430f361] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:29:02.240 INFO 3374086 --- [ocp-async-2,db07fab64bdced5a,1ecbac4593230936] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:29:02.325 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:29:04.182 INFO 3374086 --- [ocp-schedules-14,3c0fcff248eccb18,1444a01b3fa8078a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2026-01-27 14:29:04.191 WARN 3374086 --- [scanner-1877446-1,3c0fcff248eccb18,876e23110d9694ba] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 15 2026-01-27 14:29:04.200 INFO 3374086 --- [alarm-detect-8,3c0fcff248eccb18,b5755d0610996e2a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 513, data is empty. 2026-01-27 14:29:04.202 INFO 3374086 --- [alarm-detect-5,3c0fcff248eccb18,1c618b90bfd212f6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 513, data is empty. 2026-01-27 14:29:04.203 INFO 3374086 --- [alarm-detect-12,3c0fcff248eccb18,c352585356ad1f78] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 513, data is empty. 2026-01-27 14:29:04.220 INFO 3374086 --- [alarm-detect-17,3c0fcff248eccb18,672d4672559207a5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 513, data is empty. 2026-01-27 14:29:04.240 WARN 3374086 --- [ocp-async-1,fdb1bc0eb0e2707c,afc5cbc2211bbc24] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:29:04.241 INFO 3374086 --- [ocp-async-7,2ee89a55c63a2aeb,f7e6f40f5adfcf0c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:29:04.280 INFO 3374086 --- [ocp-schedules-25,13f9684aa31508db,80d223a90d428ad1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2026-01-27 14:29:04.308 INFO 3374086 --- [prometheus-minute-schedule-1,80bfa1fa552cca81,5d75204bf114f687] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495400, interval=60 2026-01-27 14:29:04.310 INFO 3374086 --- [prometheus-minute-schedule-1,80bfa1fa552cca81,5d75204bf114f687] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:29:04.446 INFO 3374086 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2026-01-27 14:29:05.326 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:29:06.241 WARN 3374086 --- [ocp-async-3,7d41a64134d83d7e,fd04d75ce6b8893d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:29:06.241 INFO 3374086 --- [ocp-async-10,fa311a92332d59a2,15d68986ca0d0702] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:29:07.040 INFO 3374086 --- [ocp-async-10,c80130a5b79f9597,cebf42fdbeddd46b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2026-01-27 14:29:07.308 INFO 3374086 --- [prometheus-minute-schedule-1,3b983081cad8f13c,d52fbd4cf0fcd701] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495400, interval=60 2026-01-27 14:29:07.311 INFO 3374086 --- [prometheus-minute-schedule-1,3b983081cad8f13c,d52fbd4cf0fcd701] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:29:08.241 INFO 3374086 --- [ocp-async-5,f3943ab33b1a1d25,c05c7b4234e3a134] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:29:08.241 WARN 3374086 --- [ocp-async-6,260fa4cd876a895c,4d20ab31aae8e848] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:29:08.327 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:29:08.854 INFO 3374086 --- [ocp-schedules-2,447c7367c18edc06,67b3b71620139298] c.o.o.s.c.scheduled.ClusterSchedules : Check all standby cluster 2026-01-27 14:29:08.867 INFO 3374086 --- [ocp-schedules-26,3920f76aa409a146,502004a5ca24c155] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2026-01-27T14:29:08.867816869+08:00, clusters=[] 2026-01-27 14:29:08.868 INFO 3374086 --- [ocp-schedules-30,4e4b23faa6b686f0,90887dd44d3295e9] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2026-01-27T14:29:08.868414397+08:00, clusters=[] 2026-01-27 14:29:08.868 INFO 3374086 --- [ocp-schedules-19,c456c24d669e21f2,b81076e1417c889f] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2026-01-27 14:29:10.002 INFO 3374086 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2026-01-27 14:29:10.002 INFO 3374086 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2026-01-27 14:29:10.241 INFO 3374086 --- [ocp-async-4,52c37f907936140e,b591275a297508e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:29:10.241 WARN 3374086 --- [ocp-async-7,76248bd2eec4dccd,926335e5dc4cb7d9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:29:10.292 INFO 3374086 --- [ocp-schedules-19,3ad90b2f683ca14d,102ce536dcf450a5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2026-01-27 14:29:10.308 INFO 3374086 --- [prometheus-minute-schedule-1,2415484332d261b4,7c47bf0db769f003] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495400, interval=60 2026-01-27 14:29:10.310 INFO 3374086 --- [prometheus-minute-schedule-1,2415484332d261b4,7c47bf0db769f003] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:29:10.454 INFO 3374086 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 2, ruleSize = 2. 2026-01-27 14:29:10.712 INFO 3374086 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1769495280, nodeCount=32, affectRows=32 2026-01-27 14:29:10.713 INFO 3374086 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1769495280, nodeCount=32, validCount=32 2026-01-27 14:29:10.715 INFO 3374086 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1769495280, nodeCount=162, affectRows=162 2026-01-27 14:29:10.715 INFO 3374086 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1769495280, nodeCount=182, affectRows=182 2026-01-27 14:29:10.716 INFO 3374086 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1769495280, nodeCount=161, validCount=161 2026-01-27 14:29:10.716 INFO 3374086 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1769495280, nodeCount=182, affectRows=182 2026-01-27 14:29:10.716 INFO 3374086 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1769495280, nodeCount=181, validCount=181 2026-01-27 14:29:10.717 INFO 3374086 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1769495280, nodeCount=216, affectRows=216 2026-01-27 14:29:10.717 INFO 3374086 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1769495280, nodeCount=181, validCount=181 2026-01-27 14:29:10.718 INFO 3374086 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1769495280, nodeCount=215, validCount=215 2026-01-27 14:29:11.327 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:29:12.241 WARN 3374086 --- [ocp-async-9,23512894c15feb5f,1ca41c707e00fda0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:29:12.241 INFO 3374086 --- [ocp-async-10,4254b22b436c096a,a2a97fc03bf98b08] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:29:13.308 INFO 3374086 --- [prometheus-minute-schedule-1,bc838a82badd0ffa,fb3dec94ff5b1108] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495400, interval=60 2026-01-27 14:29:13.310 INFO 3374086 --- [prometheus-minute-schedule-1,bc838a82badd0ffa,fb3dec94ff5b1108] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:29:14.222 INFO 3374086 --- [ocp-schedules-1,b03a018b348b95c9,f40d00c324c72b99] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2026-01-27 14:29:14.230 WARN 3374086 --- [scanner-1877655-1,b03a018b348b95c9,d973f682a17e40df] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 15 2026-01-27 14:29:14.241 WARN 3374086 --- [ocp-async-1,36a681ce85ecf6f4,c261682e48266281] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:29:14.241 INFO 3374086 --- [ocp-async-6,9f4eb20e0ce0ceb6,ff44db617a35ee75] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:29:14.328 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:29:14.421 ERROR 3374086 --- [metric-archive-async-29,,] c.o.o.m.storage.AsyncComputeDataDao : Archive node list to table metric_data_async error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_async` (`record_id`, `timestamp`, `data`, `interval`,`measurement_id`) VALUES (?, ?, ?, ?, ?)]; (conn=1800970474) Duplicate entry '872410842752393216-1769495280-872411845576925184' for key 'PRIMARY' 2026-01-27 14:29:14.421 ERROR 3374086 --- [metric-archive-async-14,,] c.o.o.m.storage.AsyncComputeDataDao : Archive node list to table metric_data_async error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_async` (`record_id`, `timestamp`, `data`, `interval`,`measurement_id`) VALUES (?, ?, ?, ?, ?)]; (conn=1800972370) Duplicate entry '872410842802724864-1769495280-872411845576925184' for key 'PRIMARY' 2026-01-27 14:29:14.453 INFO 3374086 --- [metric-archive-async-29,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1769495280, nodeCount=16, affectRows=16 2026-01-27 14:29:14.454 INFO 3374086 --- [metric-archive-async-14,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1769495280, nodeCount=16, affectRows=16 2026-01-27 14:29:16.241 INFO 3374086 --- [ocp-async-7,78ea004e0d0d3a1b,418e8f151c73df08] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:29:16.241 WARN 3374086 --- [ocp-async-3,a63b2f790593372d,ae0e041788a4286d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:29:16.308 INFO 3374086 --- [prometheus-minute-schedule-1,18657a1aec6a3040,bfcfa1878aaee9b1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495400, interval=60 2026-01-27 14:29:16.310 INFO 3374086 --- [prometheus-minute-schedule-1,18657a1aec6a3040,bfcfa1878aaee9b1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:29:17.040 INFO 3374086 --- [ocp-async-6,7eb529e9b6efcf97,6e0a6baac88e4b20] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2026-01-27 14:29:17.328 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:29:18.242 WARN 3374086 --- [ocp-async-9,7470f3d7fdd486c4,0493a9d415f48558] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:29:18.242 INFO 3374086 --- [ocp-async-4,5c6090b173f58c1d,4bca2e725194ec25] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:29:19.308 INFO 3374086 --- [prometheus-minute-schedule-1,31259845156d9fe9,5634ee49a8446748] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495400, interval=60 2026-01-27 14:29:19.310 INFO 3374086 --- [prometheus-minute-schedule-1,31259845156d9fe9,5634ee49a8446748] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:29:20.242 WARN 3374086 --- [ocp-async-10,cfb5fb7a066ba7f4,3b9dc114e102dac4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:29:20.242 INFO 3374086 --- [ocp-async-2,b47d91775255d431,fd58ed9bef370789] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:29:20.295 INFO 3374086 --- [ocp-schedules-5,4563d57498bee44b,77971da18d266780] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2026-01-27 14:29:20.329 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:29:22.242 WARN 3374086 --- [ocp-async-3,9ba173d9f2591fc8,8e70ac6c9ba75a9d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:29:22.242 INFO 3374086 --- [ocp-async-7,18248f81bef8c0bf,4868022209432fcd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:29:22.308 INFO 3374086 --- [prometheus-minute-schedule-1,f88b34208504f28a,effacd843d3c3b43] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495400, interval=60 2026-01-27 14:29:22.310 INFO 3374086 --- [prometheus-minute-schedule-1,f88b34208504f28a,effacd843d3c3b43] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:29:23.329 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:29:24.242 WARN 3374086 --- [ocp-async-8,946cae82d42c6e61,ea6fe9b77f1084b7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:29:24.242 INFO 3374086 --- [ocp-async-5,817ff956df937587,0be5fa6532824027] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:29:24.253 INFO 3374086 --- [ocp-schedules-5,7b1c391b46c6239e,55d6f93675666e6b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2026-01-27 14:29:24.266 WARN 3374086 --- [scanner-1877793-1,7b1c391b46c6239e,dc977f1e290d21fe] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 15 2026-01-27 14:29:24.273 INFO 3374086 --- [alarm-detect-19,7b1c391b46c6239e,84ea0e2db1a4f03c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 513, data is empty. 2026-01-27 14:29:24.276 INFO 3374086 --- [alarm-detect-20,7b1c391b46c6239e,9bd52a987e189fc2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 513, data is empty. 2026-01-27 14:29:24.276 INFO 3374086 --- [alarm-detect-18,7b1c391b46c6239e,9404e29703ceb590] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 513, data is empty. 2026-01-27 14:29:24.283 INFO 3374086 --- [ocp-schedules-7,ae15995ca17920a6,e0b0f6a662bab2f5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2026-01-27 14:29:24.294 INFO 3374086 --- [alarm-detect-5,7b1c391b46c6239e,cf1da172efec9a21] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 513, data is empty. 2026-01-27 14:29:25.308 INFO 3374086 --- [prometheus-minute-schedule-1,10d444d9d313644e,04b2aa1447c12334] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495400, interval=60 2026-01-27 14:29:25.311 INFO 3374086 --- [prometheus-minute-schedule-1,10d444d9d313644e,04b2aa1447c12334] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:29:26.121 INFO 3374086 --- [ocp-schedules-29,1215448f9c2d103c,41fe115756f3b3da] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2026-01-27 14:29:26.122 INFO 3374086 --- [ocp-schedules-4,25a7d824382b4819,55cc5dee62b02cd3] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2026-01-27 14:29:26.124 INFO 3374086 --- [ocp-schedules-16,cc3b73518e470de7,d4e3abefb64a1fb7] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2026-01-27 14:29:26.128 INFO 3374086 --- [ocp-schedules-8,17c2dc195e7fcc32,38b294ee3381d9ae] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2026-01-27 14:29:26.134 INFO 3374086 --- [ocp-schedules-23,94570ca1f01ba690,d05aaf6057d56a15] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.132.0.48] 2026-01-27 14:29:26.134 INFO 3374086 --- [ocp-schedules-1,2874ff78e1c51041,e0c816b1c54168bd] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.132.0.48] 2026-01-27 14:29:26.135 INFO 3374086 --- [ocp-schedules-26,30f142df379cfac6,79be4f6b35913d82] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.132.0.48] 2026-01-27 14:29:26.135 INFO 3374086 --- [host-check-host-status1,94570ca1f01ba690,bcb5ce196b86546e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.132.0.48, originalStatus=ONLINE 2026-01-27 14:29:26.135 INFO 3374086 --- [host-check-agent-status1,2874ff78e1c51041,736695ffac7fe5cb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.132.0.48, originalStatus=ONLINE 2026-01-27 14:29:26.135 INFO 3374086 --- [host-check-clockdiff1,30f142df379cfac6,20b7637b6e2293b3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.132.0.48], timeoutMillis=20000 2026-01-27 14:29:26.142 INFO 3374086 --- [host-check-agent-status1,2874ff78e1c51041,736695ffac7fe5cb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2026-01-27 14:29:26.142 INFO 3374086 --- [host-check-host-status1,94570ca1f01ba690,bcb5ce196b86546e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2026-01-27 14:29:26.151 INFO 3374086 --- [host-check-host-status1,94570ca1f01ba690,bcb5ce196b86546e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2026-01-27 14:29:26.151 INFO 3374086 --- [host-check-agent-status1,2874ff78e1c51041,736695ffac7fe5cb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2026-01-27 14:29:26.176 INFO 3374086 --- [host-check-host-status1,94570ca1f01ba690,bcb5ce196b86546e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.132.0.48:62888/api/v1/time, params:null 2026-01-27 14:29:26.176 INFO 3374086 --- [host-check-agent-status1,2874ff78e1c51041,736695ffac7fe5cb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.132.0.48:62888/api/v1/agent/status, params:null 2026-01-27 14:29:26.178 INFO 3374086 --- [host-check-host-status1,94570ca1f01ba690,bcb5ce196b86546e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.132.0.48, timestamp=2026-01-27T14:29:26.177616225+08:00 2026-01-27 14:29:26.179 INFO 3374086 --- [host-check-host-status1,94570ca1f01ba690,bcb5ce196b86546e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2026-01-27T14:29:26.178747921+08:00 2026-01-27 14:29:26.181 INFO 3374086 --- [host-check-agent-status1,2874ff78e1c51041,736695ffac7fe5cb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.132.0.48 2026-01-27 14:29:26.181 INFO 3374086 --- [host-check-agent-status1,2874ff78e1c51041,736695ffac7fe5cb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.132.0.48 2026-01-27 14:29:26.181 INFO 3374086 --- [host-check-agent-status1,2874ff78e1c51041,736695ffac7fe5cb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.132.0.48 2026-01-27 14:29:26.185 INFO 3374086 --- [host-check-host-status1,94570ca1f01ba690,bcb5ce196b86546e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2026-01-27T14:29:26.178747921+08:00 and status AVAILABLE 2026-01-27 14:29:26.187 INFO 3374086 --- [host-check-host-status1,94570ca1f01ba690,bcb5ce196b86546e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2026-01-27 14:29:26.203 INFO 3374086 --- [host-check-host-status1,94570ca1f01ba690,bcb5ce196b86546e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2026-01-27 14:29:26.211 INFO 3374086 --- [host-check-host-status1,94570ca1f01ba690,bcb5ce196b86546e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2026-01-27 14:29:26.236 INFO 3374086 --- [host-check-host-status1,94570ca1f01ba690,bcb5ce196b86546e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.132.0.48:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2026-01-27 14:29:26.242 WARN 3374086 --- [ocp-async-4,e05b387caebc4767,86d30ac42f208c0a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:29:26.242 INFO 3374086 --- [ocp-async-1,47c844ecf6e3a19c,419dbc54788cbc03] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:29:26.284 INFO 3374086 --- [host-check-host-status1,94570ca1f01ba690,bcb5ce196b86546e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2026-01-27 14:29:26.294 INFO 3374086 --- [host-check-clockdiff1,30f142df379cfac6,20b7637b6e2293b3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1769495366 0 -1 , error= 2026-01-27 14:29:26.302 INFO 3374086 --- [host-check-host-status1,94570ca1f01ba690,bcb5ce196b86546e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2026-01-27 14:29:26.319 INFO 3374086 --- [host-check-host-status1,94570ca1f01ba690,bcb5ce196b86546e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.132.0.48:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2026-01-27 14:29:26.330 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:29:26.359 INFO 3374086 --- [host-check-host-status1,94570ca1f01ba690,bcb5ce196b86546e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2026-01-27 14:29:26.377 INFO 3374086 --- [host-check-host-status1,94570ca1f01ba690,bcb5ce196b86546e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2026-01-27 14:29:26.393 INFO 3374086 --- [host-check-host-status1,94570ca1f01ba690,bcb5ce196b86546e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.132.0.48:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2026-01-27 14:29:26.433 INFO 3374086 --- [host-check-host-status1,94570ca1f01ba690,bcb5ce196b86546e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2026-01-27 14:29:26.451 INFO 3374086 --- [host-check-host-status1,94570ca1f01ba690,bcb5ce196b86546e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2026-01-27 14:29:27.040 INFO 3374086 --- [ocp-async-10,ee8099cba198dfc0,aa84aafa3af81b27] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2026-01-27 14:29:28.242 INFO 3374086 --- [ocp-async-3,6094af914998ff40,e267818e8e52d9dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:29:28.242 WARN 3374086 --- [ocp-async-6,379d018bec4603dd,31fdf798e0cd7f7b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:29:28.308 INFO 3374086 --- [prometheus-minute-schedule-1,d98822ecb63e8855,6e7a9f345be73b0e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495400, interval=60 2026-01-27 14:29:28.310 INFO 3374086 --- [prometheus-minute-schedule-1,d98822ecb63e8855,6e7a9f345be73b0e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:29:29.330 INFO 3374086 --- [ocp-schedules-19,cc4f9d78fe83d083,a747427f26b552e0] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2026-01-27 14:29:29.330 INFO 3374086 --- [ocp-schedules-19,cc4f9d78fe83d083,a747427f26b552e0] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2026-01-27 14:29:29.330 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:29:29.344 INFO 3374086 --- [Collect all tenants major compactions1,cc4f9d78fe83d083,beb4a8327a07aecf] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myocp, tenant:ocp_monitor 2026-01-27 14:29:29.358 INFO 3374086 --- [Collect all tenants major compactions1,cc4f9d78fe83d083,beb4a8327a07aecf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2026-01-27 14:29:29.360 INFO 3374086 --- [Collect all tenants major compactions1,cc4f9d78fe83d083,beb4a8327a07aecf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2026-01-27 14:29:29.363 INFO 3374086 --- [Collect all tenants major compactions1,cc4f9d78fe83d083,beb4a8327a07aecf] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myocp, tenant:ocp_monitor 2026-01-27 14:29:29.363 INFO 3374086 --- [ocp-schedules-19,cc4f9d78fe83d083,a747427f26b552e0] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2026-01-27 14:29:30.242 INFO 3374086 --- [ocp-async-8,a34ced6c92e7eaea,65ebbf0fed251a89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:29:30.242 WARN 3374086 --- [ocp-async-5,d564e4e9aa952268,e78e0a771570b36f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:29:30.258 INFO 3374086 --- [ocp-schedules-16,bd4b1f4a9c957280,c7788bd795d00b50] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2026-01-27 14:29:30.297 INFO 3374086 --- [ocp-schedules-13,3aa9a9508ef65143,a71212ca976636b1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2026-01-27 14:29:31.308 INFO 3374086 --- [prometheus-minute-schedule-1,44013cf88cc40d03,475d729e77eca1a0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495400, interval=60 2026-01-27 14:29:31.310 INFO 3374086 --- [prometheus-minute-schedule-1,44013cf88cc40d03,475d729e77eca1a0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:29:32.243 INFO 3374086 --- [ocp-async-4,735d5d79c6e7196d,463596c2c80b05ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:29:32.243 WARN 3374086 --- [ocp-async-10,54cd8bbde0429895,9df92a3af2ba4bb1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:29:32.331 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:29:34.243 INFO 3374086 --- [ocp-async-6,3d0e91b38c07bbac,dec8971676ebb705] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:29:34.243 WARN 3374086 --- [ocp-async-7,8270bbceaa7892d6,a1534a83e27adce7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:29:34.296 INFO 3374086 --- [ocp-schedules-13,0f88947cfa8c1b3f,63f1f073e21ac2ea] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2026-01-27 14:29:34.304 WARN 3374086 --- [scanner-1877954-1,0f88947cfa8c1b3f,32ec8b298d71d831] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 15 2026-01-27 14:29:34.308 INFO 3374086 --- [prometheus-minute-schedule-1,db06137bbc36d895,d4117ff03167e85a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495400, interval=60 2026-01-27 14:29:34.310 INFO 3374086 --- [prometheus-minute-schedule-1,db06137bbc36d895,d4117ff03167e85a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:29:35.331 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:29:35.712 INFO 3374086 --- [ocp-schedules-19,2becfcb94282b318,beb82eac609454dd] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2026-01-27 14:29:35.712 INFO 3374086 --- [ocp-schedules-19,2becfcb94282b318,beb82eac609454dd] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2026-01-27 14:29:35.748 INFO 3374086 --- [ocp-schedules-19,2becfcb94282b318,beb82eac609454dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2026-01-27 14:29:35.750 INFO 3374086 --- [ocp-schedules-19,2becfcb94282b318,beb82eac609454dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2026-01-27 14:29:36.243 INFO 3374086 --- [ocp-async-3,01d5d240c94caf65,a73d9002dc6d21e7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:29:36.243 WARN 3374086 --- [ocp-async-5,4ff5de9d9b12bfa5,7a3c79897bba0658] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:29:37.041 INFO 3374086 --- [ocp-async-9,c76a1ae5af6e83bf,fcb5e359fc6823a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2026-01-27 14:29:37.308 INFO 3374086 --- [prometheus-minute-schedule-1,4e2e8c16ce702b95,60b1dcb9d77b52d8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495400, interval=60 2026-01-27 14:29:37.311 INFO 3374086 --- [prometheus-minute-schedule-1,4e2e8c16ce702b95,60b1dcb9d77b52d8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:29:38.243 WARN 3374086 --- [ocp-async-10,0b04be7752a3fb29,082a6559b8246184] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:29:38.243 INFO 3374086 --- [ocp-async-10,60654728b059c4bf,d1bcc98d7ab706a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:29:38.332 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:29:40.002 INFO 3374086 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2026-01-27 14:29:40.002 INFO 3374086 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2026-01-27 14:29:40.243 WARN 3374086 --- [ocp-async-7,a8a3ea278286cd6f,76a03d24b9f4f083] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:29:40.243 INFO 3374086 --- [ocp-async-6,bf0b229f1bf30304,7bcd1dceb91ef063] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:29:40.299 INFO 3374086 --- [ocp-schedules-24,0d613f23204dcc6b,d3059a185a55da4d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2026-01-27 14:29:40.308 INFO 3374086 --- [prometheus-minute-schedule-1,56caf3903bcc84d4,3a9d9e2d0bf837a1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495400, interval=60 2026-01-27 14:29:40.311 INFO 3374086 --- [prometheus-minute-schedule-1,56caf3903bcc84d4,3a9d9e2d0bf837a1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:29:41.333 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:29:42.243 WARN 3374086 --- [ocp-async-5,a8b8bfbf407a74ca,3fc3dd4d2c76a898] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:29:42.243 INFO 3374086 --- [ocp-async-3,d02f370b588f7f9f,850cb91275c6b96c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:29:43.308 INFO 3374086 --- [prometheus-minute-schedule-1,6ed3c170b67b5d98,61ac652aed3c804b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495400, interval=60 2026-01-27 14:29:43.311 INFO 3374086 --- [prometheus-minute-schedule-1,6ed3c170b67b5d98,61ac652aed3c804b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:29:44.243 WARN 3374086 --- [ocp-async-4,ef4742e24c905411,4fa27496acd11d6d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:29:44.243 INFO 3374086 --- [ocp-async-1,fb5b7877143ead7d,2935309c12de5dcc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:29:44.286 INFO 3374086 --- [ocp-schedules-18,fd3b2dfca562d5c2,a9c999747a614e2b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2026-01-27 14:29:44.328 INFO 3374086 --- [ocp-schedules-25,dd59daf24f0c9aff,1ebbebe1b42d29a1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2026-01-27 14:29:44.333 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:29:44.338 WARN 3374086 --- [scanner-1878088-1,dd59daf24f0c9aff,a5e962418c5b6bbd] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 15 2026-01-27 14:29:44.344 INFO 3374086 --- [alarm-detect-12,dd59daf24f0c9aff,8d732ec37f570aea] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 513, data is empty. 2026-01-27 14:29:44.347 INFO 3374086 --- [alarm-detect-1,dd59daf24f0c9aff,9513199d49951c76] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 513, data is empty. 2026-01-27 14:29:44.349 INFO 3374086 --- [alarm-detect-6,dd59daf24f0c9aff,39007ffa33d68d08] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 513, data is empty. 2026-01-27 14:29:44.365 INFO 3374086 --- [alarm-detect-19,dd59daf24f0c9aff,82fd1e68f2080ff0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 513, data is empty. 2026-01-27 14:29:46.243 WARN 3374086 --- [ocp-async-7,93697f4531396774,ff7a423c035fd98c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:29:46.244 INFO 3374086 --- [ocp-async-2,e95b0c923838233e,2129b5aec6e8ea7f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:29:46.308 INFO 3374086 --- [prometheus-minute-schedule-1,e1b29ec9af83f44b,4164289c65932706] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495400, interval=60 2026-01-27 14:29:46.310 INFO 3374086 --- [prometheus-minute-schedule-1,e1b29ec9af83f44b,4164289c65932706] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:29:47.041 INFO 3374086 --- [ocp-async-10,f64215dcf0393862,8135455747e0b513] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2026-01-27 14:29:47.310 INFO 3374086 --- [prometheus-cache-monitor1,93c8a9e55e07b87b,ec736e46ad697680] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":3100,"minuteWriteQueue":0,"minuteReadCacheSize":1173,"secondWriteCacheSize":776,"seriesIdKeySize":5585,"secondWriteQueue":0,"minuteWriteCacheSize":1176} 2026-01-27 14:29:47.310 INFO 3374086 --- [prometheus-cache-monitor1,93c8a9e55e07b87b,ec736e46ad697680] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_query_rt_total_cumulative_count=29, ob_sysstat_40012=17, ob_server_resource_memory_assigned_bytes=10, node_memory_sreclaimable_bytes=12, ob_sysstat_40011=17, ob_sysstat_40010=17, plan_max_cache_item_size=9, ob_unit_num=10, tcp_retrans=13, ob_database_disk_required_size=32, go_memstats_mspan_sys_bytes=10, go_memstats_stack_sys_bytes=10, exporter_export_metrics_bytes_total=17, node_filesystem_size_bytes=45, http_server_requests_seconds_count=91, ob_tenant_request_queue_total_size=16, go_gc_gogc_percent=10, jvm_gc_memory_allocated_bytes_total=4, ob_compaction_frozen_time=12, ob_system_event_total_waits=26, ob_tenant_disk_v4_log_disk_size=12, ob_sysstat_50050=17, log_stream_degraded_count=16, ob_sysstat_40019=17, ob_sysstat_40018=17, processor_process_metrics_total=14, ob_plan_cache_hit_total=16, jvm_gc_max_data_size_bytes=4, node_filesystem_readonly=44, go_memstats_frees_total=10, io_await=35, host_available=7, ob_sql_event_in_sequence_load=16, go_memstats_mallocs_total=10, ob_unit_config_max_memory_bytes=19, ob_sysstat_200006=17, ob_database_disk_data_size=31, io_util=35, ob_sysstat_200005=17, ob_sysstat_200002=17, primary_leaders_cnt=15, ob_sysstat_200001=17, ob_tenant_memstore_freeze_cnt=10, ob_tenant_disk_v4_data_size=12, ob_query_rt_table_total_cumulative_count=29, ob_tenant_unit_min_memory=26, ob_sysstat_60087=17, ob_memstore_total_bytes=20, jvm_threads_states_threads=10, process_resident_memory_bytes=10, ob_cache_size_bytes=30, ob_tenant_assigned_cpu_total=16, ob_tenant_unit_max_memory=26, input_collect_metrics_total=15, ob_sysstat_50039=17, ob_sysstat_60083=17, ob_sysstat_50038=17, node_boot_time_seconds=12, cpu_count=13, node_disk_io_time_weighted_seconds_total=18, node_disk_write_time_seconds_total=18, output_write_metrics_total=11, ob_sysstat_50035=17, ob_cluster_sync=7, ob_sysstat_50034=17, process_open_fds=10, ob_sysstat_50037=17, ob_sysstat_50036=17, process_memory_rss_bytes=17, http_status_count=14, process_max_fds=10, ob_server_resource_cpu_assigned_percent=10, ob_sysstat_50030= 2026-01-27 14:29:47.334 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:29:48.244 WARN 3374086 --- [ocp-async-5,fc43012f56ea4b77,fc4060973321e906] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:29:48.244 INFO 3374086 --- [ocp-async-3,0216cae68ebd0003,b2ba6753715518f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:29:48.294 INFO 3374086 --- [ocp-schedules-26,ffa558968d43ce86,4a4e3b0805244e46] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.132.0.48] 2026-01-27 14:29:48.294 INFO 3374086 --- [ocp-schedules-26,ffa558968d43ce86,4a4e3b0805244e46] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2026-01-27 14:29:48.302 INFO 3374086 --- [ocp-schedules-26,ffa558968d43ce86,4a4e3b0805244e46] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2026-01-27 14:29:48.310 INFO 3374086 --- [ocp-schedules-26,ffa558968d43ce86,4a4e3b0805244e46] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2026-01-27 14:29:48.337 INFO 3374086 --- [ocp-schedules-26,ffa558968d43ce86,4a4e3b0805244e46] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.132.0.48:62889/api/v1/log/alarms, params:null 2026-01-27 14:29:48.339 INFO 3374086 --- [ocp-schedules-26,ffa558968d43ce86,4a4e3b0805244e46] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2026-01-27 14:29:49.308 INFO 3374086 --- [prometheus-minute-schedule-1,6bdd473613e93a9d,edea6d0155df92a8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495400, interval=60 2026-01-27 14:29:49.311 INFO 3374086 --- [prometheus-minute-schedule-1,6bdd473613e93a9d,edea6d0155df92a8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:29:50.244 WARN 3374086 --- [ocp-async-4,f6e87e79c77ce2e8,156ea3c55eae8b86] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:29:50.244 INFO 3374086 --- [ocp-async-8,4699180e53358e9a,f2e8d0b849734d4c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:29:50.301 INFO 3374086 --- [ocp-schedules-7,df0a7dfcf9cb5eb5,347b6af569941f46] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2026-01-27 14:29:50.334 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:29:52.244 WARN 3374086 --- [ocp-async-7,81bba0822f1ee79b,656c76c11e4a4b54] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:29:52.244 INFO 3374086 --- [ocp-async-2,5bc6bf509a24f34f,0989f4fb08f45afe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:29:52.308 INFO 3374086 --- [prometheus-minute-schedule-1,a94397792d617f89,13db0ec4fd878ecf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495400, interval=60 2026-01-27 14:29:52.310 INFO 3374086 --- [prometheus-minute-schedule-1,a94397792d617f89,13db0ec4fd878ecf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:29:53.335 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:29:54.244 WARN 3374086 --- [ocp-async-9,2fa6f02931421590,df83fe47da5b866f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:29:54.244 INFO 3374086 --- [ocp-async-5,6a2eb78c19fee823,d4884e09ebb28758] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:29:54.367 INFO 3374086 --- [ocp-schedules-10,5fef18e400b3695d,98d8cdfb2c2d5ad8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2026-01-27 14:29:54.375 WARN 3374086 --- [scanner-1878244-1,5fef18e400b3695d,5e3d20b841466d74] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 15 2026-01-27 14:29:55.308 INFO 3374086 --- [prometheus-minute-schedule-1,f04aee12d4ab10af,5f05887a48065921] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495400, interval=60 2026-01-27 14:29:55.311 INFO 3374086 --- [prometheus-minute-schedule-1,f04aee12d4ab10af,5f05887a48065921] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:29:56.121 INFO 3374086 --- [ocp-schedules-4,2c2c5f1b69335a0e,04e39eccc41484dd] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2026-01-27 14:29:56.122 INFO 3374086 --- [ocp-schedules-7,c98e4b4f80bf0d41,18a294373c3c6142] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2026-01-27 14:29:56.124 INFO 3374086 --- [ocp-schedules-21,574fb50f01444868,a8ec8547b142acca] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2026-01-27 14:29:56.128 INFO 3374086 --- [ocp-schedules-26,2dc89b4363e5b046,8daba492d5421168] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2026-01-27 14:29:56.130 INFO 3374086 --- [ocp-schedules-6,02e79614bbd18b08,b8a937e062507d4b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2026-01-27 14:29:56.131 INFO 3374086 --- [ocp-schedules-3,bb5914e52b1c3da8,06c21e322787fe0a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2026-01-27 14:29:56.133 INFO 3374086 --- [ocp-schedules-6,02e79614bbd18b08,b8a937e062507d4b] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2026-01-27 14:29:56.134 INFO 3374086 --- [ocp-schedules-29,d88b1ccd6e7614a7,99611f603bee6654] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.132.0.48] 2026-01-27 14:29:56.134 INFO 3374086 --- [ocp-schedules-13,5a269a9e75342dd3,7208fd745b80129a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.132.0.48] 2026-01-27 14:29:56.135 INFO 3374086 --- [host-check-clockdiff1,d88b1ccd6e7614a7,b39b901f088977a7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.132.0.48], timeoutMillis=20000 2026-01-27 14:29:56.135 INFO 3374086 --- [ocp-schedules-9,5b4a517874d6d7a2,c89d86da924901c9] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.132.0.48] 2026-01-27 14:29:56.135 INFO 3374086 --- [host-check-host-status1,5a269a9e75342dd3,4b485d31ea4b8800] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.132.0.48, originalStatus=ONLINE 2026-01-27 14:29:56.135 INFO 3374086 --- [host-check-agent-status1,5b4a517874d6d7a2,fb714c7243468656] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.132.0.48, originalStatus=ONLINE 2026-01-27 14:29:56.138 INFO 3374086 --- [ocp-schedules-3,bb5914e52b1c3da8,06c21e322787fe0a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2026-01-27 14:29:56.142 INFO 3374086 --- [host-check-host-status1,5a269a9e75342dd3,4b485d31ea4b8800] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2026-01-27 14:29:56.143 INFO 3374086 --- [host-check-agent-status1,5b4a517874d6d7a2,fb714c7243468656] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2026-01-27 14:29:56.151 INFO 3374086 --- [host-check-host-status1,5a269a9e75342dd3,4b485d31ea4b8800] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2026-01-27 14:29:56.151 INFO 3374086 --- [host-check-agent-status1,5b4a517874d6d7a2,fb714c7243468656] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2026-01-27 14:29:56.163 INFO 3374086 --- [ocp-schedules-28,8e7439156252324a,ad6fcef8c12f061e] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2026-01-27 14:29:56.163 INFO 3374086 --- [ocp-schedules-28,8e7439156252324a,ad6fcef8c12f061e] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2026-01-27 14:29:56.177 INFO 3374086 --- [host-check-host-status1,5a269a9e75342dd3,4b485d31ea4b8800] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.132.0.48:62888/api/v1/time, params:null 2026-01-27 14:29:56.177 INFO 3374086 --- [host-check-agent-status1,5b4a517874d6d7a2,fb714c7243468656] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.132.0.48:62888/api/v1/agent/status, params:null 2026-01-27 14:29:56.179 INFO 3374086 --- [host-check-host-status1,5a269a9e75342dd3,4b485d31ea4b8800] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.132.0.48, timestamp=2026-01-27T14:29:56.178225375+08:00 2026-01-27 14:29:56.179 INFO 3374086 --- [host-check-host-status1,5a269a9e75342dd3,4b485d31ea4b8800] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2026-01-27T14:29:56.179341631+08:00 2026-01-27 14:29:56.182 INFO 3374086 --- [host-check-agent-status1,5b4a517874d6d7a2,fb714c7243468656] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.132.0.48 2026-01-27 14:29:56.182 INFO 3374086 --- [host-check-agent-status1,5b4a517874d6d7a2,fb714c7243468656] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.132.0.48 2026-01-27 14:29:56.182 INFO 3374086 --- [host-check-agent-status1,5b4a517874d6d7a2,fb714c7243468656] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.132.0.48 2026-01-27 14:29:56.185 INFO 3374086 --- [host-check-host-status1,5a269a9e75342dd3,4b485d31ea4b8800] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2026-01-27T14:29:56.179341631+08:00 and status AVAILABLE 2026-01-27 14:29:56.187 INFO 3374086 --- [host-check-host-status1,5a269a9e75342dd3,4b485d31ea4b8800] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2026-01-27 14:29:56.199 INFO 3374086 --- [pool-overview-container-7,b426b050d5d5b6ac,407275dc51eb26fc] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2026-01-27 14:29:56.199 INFO 3374086 --- [pool-overview-container-4,68972fca9dce376f,5a2e80623f12c75b] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2026-01-27 14:29:56.203 INFO 3374086 --- [host-check-host-status1,5a269a9e75342dd3,4b485d31ea4b8800] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2026-01-27 14:29:56.211 INFO 3374086 --- [host-check-host-status1,5a269a9e75342dd3,4b485d31ea4b8800] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2026-01-27 14:29:56.212 INFO 3374086 --- [pool-14-thread-10,68972fca9dce376f,63e8601e32982ab7] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2026-01-27 14:29:56.212 INFO 3374086 --- [pool-14-thread-10,68972fca9dce376f,63e8601e32982ab7] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2026-01-27 14:29:56.212 INFO 3374086 --- [pool-14-thread-9,68972fca9dce376f,71521cf29b8c238b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2026-01-27 14:29:56.212 INFO 3374086 --- [pool-14-thread-12,68972fca9dce376f,83596a8c48db528e] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2026-01-27 14:29:56.212 INFO 3374086 --- [pool-14-thread-11,68972fca9dce376f,fee8732ebdbacd72] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2026-01-27 14:29:56.215 WARN 3374086 --- [host-check-clockdiff1,d88b1ccd6e7614a7,b39b901f088977a7] c.o.ocp.core.task.util.ClockDiff : Clock-diff failed. args=[10.132.0.48], exception={}. Please try other ocp.host.check.clock-diff.mode in System Parameters org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1) at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404) at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:166) at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:153) at com.oceanbase.ocp.core.util.LocalCommandExecutor.exec(LocalCommandExecutor.java:50) at com.oceanbase.ocp.core.task.util.ClockDiff.execute(ClockDiff.java:95) at com.oceanbase.ocp.core.task.util.ClockDiff.diffWithIcmpTimestamp(ClockDiff.java:52) at com.oceanbase.ocp.core.task.util.ClockDiff$ClockDiffMode.doRun(ClockDiff.java:172) at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.doCheckHostClockDiff(HostCheckManagerImpl.java:498) at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostClockDiff(HostCheckManagerImpl.java:486) at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$doExecute$0(DistributedExclusiveExecutor.java:73) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) 2026-01-27 14:29:56.215 ERROR 3374086 --- [host-check-clockdiff1,d88b1ccd6e7614a7,b39b901f088977a7] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.UnexpectedException occurred with code error.common.unexpected, and args [Execute clock diff failed., 10.132.0.48] 2026-01-27 14:29:56.215 WARN 3374086 --- [host-check-clockdiff1,d88b1ccd6e7614a7,b39b901f088977a7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host clock diff fail. hostId=3, ip=10.132.0.48, exception={} com.oceanbase.ocp.core.exception.UnexpectedException: [OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=COMMON_UNEXPECTED, args=Execute clock diff failed.,10.132.0.48 at jdk.internal.reflect.GeneratedConstructorAccessor545.newInstance(Unknown Source) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Unknown Source) at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.core.util.ExceptionUtils.throwException(ExceptionUtils.java:90) at com.oceanbase.ocp.core.util.ExceptionUtils.unExpected(ExceptionUtils.java:77) at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.doCheckHostClockDiff(HostCheckManagerImpl.java:500) at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostClockDiff(HostCheckManagerImpl.java:486) at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$doExecute$0(DistributedExclusiveExecutor.java:73) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) 2026-01-27 14:29:56.216 INFO 3374086 --- [ForkJoinPool-6-worker-379,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2026-01-27 14:29:56.216 INFO 3374086 --- [ForkJoinPool-6-worker-379,68972fca9dce376f,1bd507af636df3f6] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of myocp 2026-01-27 14:29:56.216 INFO 3374086 --- [ForkJoinPool-6-worker-379,68972fca9dce376f,1bd507af636df3f6] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of myocp 2026-01-27 14:29:56.217 INFO 3374086 --- [ForkJoinPool-6-worker-380,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2026-01-27 14:29:56.218 INFO 3374086 --- [pool-14-thread-11,68972fca9dce376f,fee8732ebdbacd72] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2026-01-27 14:29:56.218 INFO 3374086 --- [pool-14-thread-11,68972fca9dce376f,fee8732ebdbacd72] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2026-01-27 14:29:56.219 INFO 3374086 --- [pool-14-thread-10,68972fca9dce376f,63e8601e32982ab7] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2026-01-27 14:29:56.220 INFO 3374086 --- [pool-14-thread-10,68972fca9dce376f,63e8601e32982ab7] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2026-01-27 14:29:56.234 WARN 3374086 --- [ForkJoinPool-6-worker-379,68972fca9dce376f,1bd507af636df3f6] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :myocp com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=myocp,ocp_meta,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:169) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:146) at jdk.internal.reflect.GeneratedMethodAccessor921.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:724) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$SpringCGLIB$$0.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:196) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:243) at java.base/java.lang.Iterable.forEach(Unknown Source) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:215) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:71) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$getClusterSessionStat$8(OasStatServiceImpl.java:280) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source) at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source) at java.base/java.util.stream.ForEachOps$ForEachTask.compute(Unknown Source) at java.base/java.util.concurrent.CountedCompleter.exec(Unknown Source) at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source) at java.base/java.util.concurrent.ForkJoinTask.invoke(Unknown Source) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(Unknown Source) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(Unknown Source) at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source) at java.base/java.util.stream.ReferencePipeline.forEach(Unknown Source) at java.base/java.util.stream.ReferencePipeline$Head.forEach(Unknown Source) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$getClusterSessionStat$9(OasStatServiceImpl.java:275) at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(Unknown Source) at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source) at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source) at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source) 2026-01-27 14:29:56.234 INFO 3374086 --- [pool-14-thread-9,68972fca9dce376f,71521cf29b8c238b] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2026-01-27 14:29:56.237 INFO 3374086 --- [pool-14-thread-12,68972fca9dce376f,83596a8c48db528e] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2026-01-27 14:29:56.237 INFO 3374086 --- [host-check-host-status1,5a269a9e75342dd3,4b485d31ea4b8800] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.132.0.48:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2026-01-27 14:29:56.244 INFO 3374086 --- [ocp-async-8,758bb93ea14677c0,a82bdec56c551ecc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:29:56.244 WARN 3374086 --- [ocp-async-7,8f870deb052d4cec,4816a79fea953de5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:29:56.287 INFO 3374086 --- [host-check-host-status1,5a269a9e75342dd3,4b485d31ea4b8800] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2026-01-27 14:29:56.306 INFO 3374086 --- [host-check-host-status1,5a269a9e75342dd3,4b485d31ea4b8800] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2026-01-27 14:29:56.322 INFO 3374086 --- [host-check-host-status1,5a269a9e75342dd3,4b485d31ea4b8800] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.132.0.48:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2026-01-27 14:29:56.335 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:29:56.363 INFO 3374086 --- [host-check-host-status1,5a269a9e75342dd3,4b485d31ea4b8800] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2026-01-27 14:29:56.373 INFO 3374086 --- [ocp-schedules-25,a54510c919923ffb,0ca1fd019ed27128] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":3100,"minuteWriteQueue":0,"minuteReadCacheSize":1173,"secondWriteCacheSize":776,"seriesIdKeySize":5585,"secondWriteQueue":0,"minuteWriteCacheSize":1176} 2026-01-27 14:29:56.373 INFO 3374086 --- [ocp-schedules-25,a54510c919923ffb,0ca1fd019ed27128] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_query_rt_total_cumulative_count=29, ob_sysstat_40012=17, ob_server_resource_memory_assigned_bytes=10, node_memory_sreclaimable_bytes=12, ob_sysstat_40011=17, ob_sysstat_40010=17, plan_max_cache_item_size=9, ob_unit_num=10, tcp_retrans=13, ob_database_disk_required_size=32, go_memstats_mspan_sys_bytes=10, go_memstats_stack_sys_bytes=10, exporter_export_metrics_bytes_total=17, node_filesystem_size_bytes=45, http_server_requests_seconds_count=91, ob_tenant_request_queue_total_size=16, go_gc_gogc_percent=10, jvm_gc_memory_allocated_bytes_total=4, ob_compaction_frozen_time=12, ob_system_event_total_waits=26, ob_tenant_disk_v4_log_disk_size=12, ob_sysstat_50050=17, log_stream_degraded_count=16, ob_sysstat_40019=17, ob_sysstat_40018=17, processor_process_metrics_total=14, ob_plan_cache_hit_total=16, jvm_gc_max_data_size_bytes=4, node_filesystem_readonly=44, go_memstats_frees_total=10, io_await=35, host_available=7, ob_sql_event_in_sequence_load=16, go_memstats_mallocs_total=10, ob_unit_config_max_memory_bytes=19, ob_sysstat_200006=17, ob_database_disk_data_size=31, io_util=35, ob_sysstat_200005=17, ob_sysstat_200002=17, primary_leaders_cnt=15, ob_sysstat_200001=17, ob_tenant_memstore_freeze_cnt=10, ob_tenant_disk_v4_data_size=12, ob_query_rt_table_total_cumulative_count=29, ob_tenant_unit_min_memory=26, ob_sysstat_60087=17, ob_memstore_total_bytes=20, jvm_threads_states_threads=10, process_resident_memory_bytes=10, ob_cache_size_bytes=30, ob_tenant_assigned_cpu_total=16, ob_tenant_unit_max_memory=26, input_collect_metrics_total=15, ob_sysstat_50039=17, ob_sysstat_60083=17, ob_sysstat_50038=17, node_boot_time_seconds=12, cpu_count=13, node_disk_io_time_weighted_seconds_total=18, node_disk_write_time_seconds_total=18, output_write_metrics_total=11, ob_sysstat_50035=17, ob_cluster_sync=7, ob_sysstat_50034=17, process_open_fds=10, ob_sysstat_50037=17, ob_sysstat_50036=17, process_memory_rss_bytes=17, http_status_count=14, process_max_fds=10, ob_server_resource_cpu_assigned_percent=10, ob_sysstat_50030= 2026-01-27 14:29:56.381 INFO 3374086 --- [host-check-host-status1,5a269a9e75342dd3,4b485d31ea4b8800] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2026-01-27 14:29:56.396 INFO 3374086 --- [ocp-schedules-30,ebdca72bfc51c209,ff5758febeb68324] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2026-01-27 14:29:56.398 INFO 3374086 --- [host-check-host-status1,5a269a9e75342dd3,4b485d31ea4b8800] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.132.0.48:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2026-01-27 14:29:56.438 INFO 3374086 --- [host-check-host-status1,5a269a9e75342dd3,4b485d31ea4b8800] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2026-01-27 14:29:56.444 WARN 3374086 --- [ocp-schedules-27,92573c3167aa903d,0d71f60710f5d33a] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=checkContingencyTenantSqlPortListen, confName=checkContingencyTenantSqlPortListen 2026-01-27 14:29:56.444 ERROR 3374086 --- [ocp-schedules-27,92573c3167aa903d,0d71f60710f5d33a] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.config.Task$OutcomeTrackingRunnable.run(Task.java:85) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.lang.reflect.InvocationTargetException: null at jdk.internal.reflect.GeneratedMethodAccessor981.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'com.oceanbase.ocp.selfcure.contingency.ContingencyInternalService' available at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:394) at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:385) at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1290) at com.oceanbase.ocp.core.util.BeanUtils.getBean(BeanUtils.java:33) at com.oceanbase.ocp.service.task.util.Beans.contingencyInternalService(Beans.java:400) at com.oceanbase.ocp.service.task.business.tenant.CheckContingencyTenantSqlPortListenTask.getContingencyParticipantTargetMatcher(CheckContingencyTenantSqlPortListenTask.java:214) at com.oceanbase.ocp.service.task.business.tenant.CheckContingencyTenantSqlPortListenTask.getContingencyParticipantCluster(CheckContingencyTenantSqlPortListenTask.java:188) at com.oceanbase.ocp.service.task.business.tenant.CheckContingencyTenantSqlPortListenTask.getAllShardingClusters(CheckContingencyTenantSqlPortListenTask.java:65) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:63) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.checkContingencyTenantSqlPortListen(TenantSchedules.java:96) ... 14 common frames omitted 2026-01-27 14:29:56.456 INFO 3374086 --- [host-check-host-status1,5a269a9e75342dd3,4b485d31ea4b8800] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2026-01-27 14:29:57.041 INFO 3374086 --- [ocp-async-1,ce07e106985ffb4a,9b195b64309c5fb8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2026-01-27 14:29:57.584 INFO 3374086 --- [ocp-schedules-30,73043f012da83dd7,28c63c67a7d1fece] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2026-01-27 14:29:58.244 INFO 3374086 --- [ocp-async-2,f0d980dd5b0b8563,8e3a17b917aefed2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:29:58.244 WARN 3374086 --- [ocp-async-6,038d60745aedbc06,fdbd6bb10af6d0db] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:29:58.270 INFO 3374086 --- [ocp-schedules-27,adac4ee6077859ed,8ca0586fa881a02b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2026-01-27 14:29:58.308 INFO 3374086 --- [prometheus-minute-schedule-1,64f5adbbec67230e,64017b141641487e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495400, interval=60 2026-01-27 14:29:58.311 INFO 3374086 --- [prometheus-minute-schedule-1,64f5adbbec67230e,64017b141641487e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:29:58.562 INFO 3374086 --- [ocp-schedules-5,aedba403703efe32,2cd994fef5118fcc] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2026-01-27 14:29:58.562 INFO 3374086 --- [ocp-schedules-5,aedba403703efe32,2cd994fef5118fcc] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=872411846587924480 2026-01-27 14:29:58.564 INFO 3374086 --- [ocp-schedules-5,aedba403703efe32,2cd994fef5118fcc] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2026-01-27 14:29:58.565 INFO 3374086 --- [ocp-schedules-5,aedba403703efe32,2cd994fef5118fcc] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=872606229282988032 2026-01-27 14:29:58.565 INFO 3374086 --- [ocp-schedules-5,aedba403703efe32,2cd994fef5118fcc] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2026-01-27 14:29:58.565 INFO 3374086 --- [ocp-schedules-5,aedba403703efe32,2cd994fef5118fcc] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=872646545121779712 2026-01-27 14:29:58.641 INFO 3374086 --- [ocp-schedules-11,2cb8f22d64221c1f,fde638500100e3a9] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2026-01-27 14:29:58.641 INFO 3374086 --- [ocp-schedules-11,2cb8f22d64221c1f,fde638500100e3a9] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2026-01-27 14:29:58.654 INFO 3374086 --- [ocp-schedules-11,2cb8f22d64221c1f,fde638500100e3a9] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2026-01-27 14:29:59.028 INFO 3374086 --- [ocp-schedules-19,87dc533e21d4f267,0e7cda96a2d501ec] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2026-01-27 14:29:59.028 INFO 3374086 --- [ocp-schedules-29,b5699dc192003ff5,1c097d3ee4014e1b] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2026-01-27 14:29:59.028 INFO 3374086 --- [ocp-schedules-13,60dfce2be16d6c1a,6f05ca6247754728] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2026-01-27 14:29:59.336 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:30:00.102 INFO 3374086 --- [prometheus-second-schedule-1,c2b04bc13277d0e9,ade42085ad4d5d8a] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=1, collectTaskCount=1 2026-01-27 14:30:00.244 INFO 3374086 --- [ocp-async-3,68e715b0634ce88f,c7ec8902c7b23080] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:30:00.245 WARN 3374086 --- [ocp-async-4,29d9434798b31d69,78c23f7207559661] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:30:00.261 INFO 3374086 --- [ocp-schedules-11,128423517d84f88f,e497accc3c9f22fb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2026-01-27 14:30:00.304 INFO 3374086 --- [ocp-schedules-21,0a8bdeb14ad59ded,18aeda1265fc13a6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2026-01-27 14:30:00.314 INFO 3374086 --- [ocp-schedules-15,ba879bb5b8e9ee1e,ccf100ce539d179f] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.132.0.48] 2026-01-27 14:30:00.314 INFO 3374086 --- [ocp-schedules-15,ba879bb5b8e9ee1e,ccf100ce539d179f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2026-01-27 14:30:00.367 INFO 3374086 --- [ocp-schedules-13,9392690836b24d06,e774b1d099dc903a] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2026-01-27 14:30:00.367 INFO 3374086 --- [ocp-schedules-13,9392690836b24d06,e774b1d099dc903a] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2026-01-27 14:30:00.368 INFO 3374086 --- [ocp-schedules-19,50d53492bc737c42,26ddb25baa5f3a5a] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant tde info status 2026-01-27 14:30:00.368 INFO 3374086 --- [ocp-schedules-19,50d53492bc737c42,26ddb25baa5f3a5a] .t.b.t.CheckAllTenantTdeInfoShardingTask : Begin to check all tenant TDE info status 2026-01-27 14:30:00.372 INFO 3374086 --- [ocp-schedules-19,50d53492bc737c42,26ddb25baa5f3a5a] .t.b.t.CheckAllTenantTdeInfoShardingTask : Check TDE info for 0 primary tenants with supported OB version 2026-01-27 14:30:00.372 INFO 3374086 --- [ocp-schedules-19,50d53492bc737c42,26ddb25baa5f3a5a] .t.b.t.CheckAllTenantTdeInfoShardingTask : Finish to check all tenant TDE info status 2026-01-27 14:30:01.308 INFO 3374086 --- [prometheus-minute-schedule-1,bac6b99ac9e2f20f,3901136337620ea0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495460, interval=60 2026-01-27 14:30:01.310 INFO 3374086 --- [prometheus-minute-schedule-1,bac6b99ac9e2f20f,3901136337620ea0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:30:01.628 INFO 3374086 --- [subtask-coordinator3,d55e43dd1f0b2a8c,7bc0428661e7a9e9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 561, operation:EXECUTE, state: RUNNING 2026-01-27 14:30:01.646 INFO 3374086 --- [subtask-executor31,063cf55febc37d7d,556bedd2455f86d9] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2026-01-27 14:30:01.647 INFO 3374086 --- [subtask-executor31,063cf55febc37d7d,556bedd2455f86d9] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=561, context=Context{parallelIdx=-1, stringMap={task_instance_id=495, task_operation=execute, latest_execution_start_time=2026-01-27T14:30:01.627040837+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=561}, listMap={}}, executor=10.132.0.47 2026-01-27 14:30:01.659 INFO 3374086 --- [subtask-executor31,063cf55febc37d7d,556bedd2455f86d9] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myocp 2026-01-27 14:30:01.700 INFO 3374086 --- [subtask-executor31,063cf55febc37d7d,556bedd2455f86d9] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myocp 2026-01-27 14:30:01.721 INFO 3374086 --- [subtask-executor31,063cf55febc37d7d,556bedd2455f86d9] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myocp 2026-01-27 14:30:01.729 INFO 3374086 --- [subtask-executor31,063cf55febc37d7d,556bedd2455f86d9] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for myocp 2026-01-27 14:30:01.730 INFO 3374086 --- [subtask-executor31,063cf55febc37d7d,556bedd2455f86d9] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myocp 2026-01-27 14:30:01.752 INFO 3374086 --- [subtask-executor31,063cf55febc37d7d,556bedd2455f86d9] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myocp 2026-01-27 14:30:01.773 INFO 3374086 --- [subtask-executor31,063cf55febc37d7d,556bedd2455f86d9] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myocp 2026-01-27 14:30:01.809 INFO 3374086 --- [subtask-executor31,063cf55febc37d7d,556bedd2455f86d9] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myocp 2026-01-27 14:30:01.835 INFO 3374086 --- [subtask-executor31,063cf55febc37d7d,556bedd2455f86d9] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myocp 2026-01-27 14:30:01.887 INFO 3374086 --- [subtask-executor31,063cf55febc37d7d,556bedd2455f86d9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 561, operation:EXECUTE, state: SUCCESSFUL 2026-01-27 14:30:01.888 INFO 3374086 --- [ocp-schedules-6,5e9d9e5a5d587a4c,02329d27cef6204b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2026-01-27 14:30:01.893 INFO 3374086 --- [ocp-schedules-6,5e9d9e5a5d587a4c,02329d27cef6204b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2026-01-27 14:30:01.893 INFO 3374086 --- [ocp-schedules-6,5e9d9e5a5d587a4c,02329d27cef6204b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2026-01-27 14:30:02.245 INFO 3374086 --- [ocp-async-1,54642dd249db8d82,227cbfc40c2c4c9b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:30:02.245 WARN 3374086 --- [ocp-async-8,43a0678ed03eaa01,ad03e781ea604d73] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:30:02.299 INFO 3374086 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:495, status:SUCCESS 2026-01-27 14:30:02.300 INFO 3374086 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:495, status:SUCCESS 2026-01-27 14:30:02.336 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:30:03.589 INFO 3374086 --- [ocp-schedules-17,50be14534f2b6c2b,7b8b53692b6b2b30] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=5585 2026-01-27 14:30:03.660 INFO 3374086 --- [ocp-schedules-17,50be14534f2b6c2b,7b8b53692b6b2b30] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=5585 2026-01-27 14:30:04.245 WARN 3374086 --- [ocp-async-6,288537e7aedf1f74,d413b014e859034f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:30:04.245 INFO 3374086 --- [ocp-async-9,e597ec7cb7784b8a,d15cf0f1842ce25b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:30:04.288 INFO 3374086 --- [ocp-schedules-2,3d98d9a3b374c143,da05f08da07ee6c3] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2026-01-27 14:30:04.308 INFO 3374086 --- [prometheus-minute-schedule-1,bc12149d4f62324d,0888bda79fb6cee8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495460, interval=60 2026-01-27 14:30:04.311 INFO 3374086 --- [prometheus-minute-schedule-1,bc12149d4f62324d,0888bda79fb6cee8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:30:04.399 INFO 3374086 --- [ocp-schedules-4,4700c0c75ecdf826,54221f897bd2f5b1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2026-01-27 14:30:04.408 WARN 3374086 --- [scanner-1878463-1,4700c0c75ecdf826,fa0f640bd15a2f1b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 15 2026-01-27 14:30:04.417 INFO 3374086 --- [alarm-detect-1,4700c0c75ecdf826,6d61cf5dcf3ef872] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 513, data is empty. 2026-01-27 14:30:04.418 INFO 3374086 --- [alarm-detect-6,4700c0c75ecdf826,69c803eb1901500d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 513, data is empty. 2026-01-27 14:30:04.419 INFO 3374086 --- [alarm-detect-15,4700c0c75ecdf826,e62b64d01aa466b4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 513, data is empty. 2026-01-27 14:30:04.436 INFO 3374086 --- [alarm-detect-6,4700c0c75ecdf826,4785d90b1874ed1c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 513, data is empty. 2026-01-27 14:30:04.446 INFO 3374086 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2026-01-27 14:30:05.257 INFO 3374086 --- [ocp-schedules-14,61dd06d7737622f0,4f4fc21755931b7e] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2026-01-27 14:30:05.257 INFO 3374086 --- [ocp-schedules-14,61dd06d7737622f0,4f4fc21755931b7e] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2026-01-27 14:30:05.265 INFO 3374086 --- [ocp-schedules-14,61dd06d7737622f0,4f4fc21755931b7e] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2026-01-27 14:30:05.265 INFO 3374086 --- [ocp-schedules-14,61dd06d7737622f0,4f4fc21755931b7e] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2026-01-27 14:30:05.265 INFO 3374086 --- [ocp-schedules-14,61dd06d7737622f0,4f4fc21755931b7e] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2026-01-27 14:30:05.271 INFO 3374086 --- [ocp-schedules-14,61dd06d7737622f0,4f4fc21755931b7e] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2026-01-27 14:30:05.271 INFO 3374086 --- [ocp-schedules-14,61dd06d7737622f0,4f4fc21755931b7e] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2026-01-27 14:30:05.271 INFO 3374086 --- [ocp-schedules-14,61dd06d7737622f0,4f4fc21755931b7e] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2026-01-27 14:30:05.274 INFO 3374086 --- [ocp-schedules-14,61dd06d7737622f0,4f4fc21755931b7e] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2026-01-27 14:30:05.274 INFO 3374086 --- [ocp-schedules-14,61dd06d7737622f0,4f4fc21755931b7e] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2026-01-27 14:30:05.274 INFO 3374086 --- [ocp-schedules-14,61dd06d7737622f0,4f4fc21755931b7e] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2026-01-27 14:30:05.285 INFO 3374086 --- [ocp-schedules-14,61dd06d7737622f0,4f4fc21755931b7e] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2026-01-27 14:30:05.337 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:30:06.245 INFO 3374086 --- [ocp-async-4,4e031c752afe4f42,696ad180c62d7cdc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:30:06.245 WARN 3374086 --- [ocp-async-5,c129d2f9e9663b29,0981ca58e669ebfa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:30:07.041 INFO 3374086 --- [ocp-async-4,6847aed436d09b5f,16e314562607e6e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2026-01-27 14:30:07.308 INFO 3374086 --- [prometheus-minute-schedule-1,38a8cc4c6e4385d8,3f883410a7703f60] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495460, interval=60 2026-01-27 14:30:07.311 INFO 3374086 --- [prometheus-minute-schedule-1,38a8cc4c6e4385d8,3f883410a7703f60] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:30:08.245 WARN 3374086 --- [ocp-async-1,f373359ce1a9ff7a,a7e3172b4cad2491] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:30:08.245 INFO 3374086 --- [ocp-async-7,242b8faeddfa0300,74545a04a2359b40] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:30:08.337 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:30:08.867 INFO 3374086 --- [ocp-schedules-23,041931e2f184a1ad,10c20039cd6676b8] c.o.o.s.c.scheduled.ClusterSchedules : Check all standby cluster 2026-01-27 14:30:08.881 INFO 3374086 --- [ocp-schedules-6,46eca727dbbfc7d6,6dbb92852b3d5f0a] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2026-01-27T14:30:08.881197252+08:00, clusters=[] 2026-01-27 14:30:08.881 INFO 3374086 --- [ocp-schedules-30,e8b24ee20033b5a3,f361c5ae6340d135] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2026-01-27T14:30:08.881198896+08:00, clusters=[] 2026-01-27 14:30:08.882 INFO 3374086 --- [ocp-schedules-22,111976f3ee9d4440,4c19101d48a53224] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2026-01-27 14:30:10.002 INFO 3374086 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2026-01-27 14:30:10.002 INFO 3374086 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2026-01-27 14:30:10.245 WARN 3374086 --- [ocp-async-5,80fcb0c618cc0bb9,7bec5575ab6cbae2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:30:10.245 INFO 3374086 --- [ocp-async-10,67c44d456f55a8a6,6ee39a303f263cc5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:30:10.306 INFO 3374086 --- [ocp-schedules-13,f46e0a90ecfb4ec0,0f6ea2a7d33f064a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2026-01-27 14:30:10.308 INFO 3374086 --- [prometheus-minute-schedule-1,393dbbad280ea54b,1b6f05e0d0848309] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495460, interval=60 2026-01-27 14:30:10.310 INFO 3374086 --- [prometheus-minute-schedule-1,393dbbad280ea54b,1b6f05e0d0848309] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:30:10.454 INFO 3374086 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 2, ruleSize = 2. 2026-01-27 14:30:10.713 INFO 3374086 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1769495340, nodeCount=1, affectRows=1 2026-01-27 14:30:10.713 INFO 3374086 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1769495340, nodeCount=1, validCount=1 2026-01-27 14:30:10.716 INFO 3374086 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1769495340, nodeCount=91, affectRows=91 2026-01-27 14:30:10.717 INFO 3374086 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1769495340, nodeCount=90, validCount=90 2026-01-27 14:30:10.717 INFO 3374086 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1769495340, nodeCount=113, affectRows=113 2026-01-27 14:30:10.717 INFO 3374086 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1769495340, nodeCount=123, affectRows=123 2026-01-27 14:30:10.717 INFO 3374086 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1769495340, nodeCount=95, affectRows=95 2026-01-27 14:30:10.717 INFO 3374086 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1769495340, nodeCount=109, affectRows=109 2026-01-27 14:30:10.717 INFO 3374086 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1769495340, nodeCount=98, affectRows=98 2026-01-27 14:30:10.718 INFO 3374086 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1769495340, nodeCount=113, validCount=113 2026-01-27 14:30:10.718 INFO 3374086 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1769495340, nodeCount=94, validCount=94 2026-01-27 14:30:10.718 INFO 3374086 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1769495340, nodeCount=146, affectRows=146 2026-01-27 14:30:10.718 INFO 3374086 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1769495340, nodeCount=98, validCount=98 2026-01-27 14:30:10.718 INFO 3374086 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1769495340, nodeCount=123, validCount=123 2026-01-27 14:30:10.718 INFO 3374086 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1769495340, nodeCount=108, validCount=108 2026-01-27 14:30:10.718 INFO 3374086 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1769495340, nodeCount=146, validCount=146 2026-01-27 14:30:11.338 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:30:12.245 INFO 3374086 --- [ocp-async-4,088d1a3ca06fd5f5,7ac2396cbaeff67b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:30:12.245 WARN 3374086 --- [ocp-async-3,1740b2314d17a7f8,04691d370af597f9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:30:12.780 INFO 3374086 --- [ocp-updater-0,,] c.o.ocp.obsdk.connector.ObVersionHolder : Invalid expired ob version cache 2026-01-27 14:30:13.308 INFO 3374086 --- [prometheus-minute-schedule-1,87b071c946887b80,7146ef25be87a974] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495460, interval=60 2026-01-27 14:30:13.311 INFO 3374086 --- [prometheus-minute-schedule-1,87b071c946887b80,7146ef25be87a974] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:30:14.245 INFO 3374086 --- [ocp-async-6,66f2c5b94ebc2d82,c40f013bf58723f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:30:14.245 WARN 3374086 --- [ocp-async-1,c078d9b853cbb843,e67e98a545f5b83a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:30:14.338 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:30:14.422 ERROR 3374086 --- [metric-archive-async-46,,] c.o.o.m.storage.AsyncComputeDataDao : Archive node list to table metric_data_async error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_async` (`record_id`, `timestamp`, `data`, `interval`,`measurement_id`) VALUES (?, ?, ?, ?, ?)]; (conn=1800974526) Duplicate entry '872410842777559040-1769495340-872411845576925184' for key 'PRIMARY' 2026-01-27 14:30:14.423 ERROR 3374086 --- [metric-archive-async-34,,] c.o.o.m.storage.AsyncComputeDataDao : Archive node list to table metric_data_async error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_async` (`record_id`, `timestamp`, `data`, `interval`,`measurement_id`) VALUES (?, ?, ?, ?, ?)]; (conn=1800958285) Duplicate entry '872410842802724864-1769495340-872411845576925184' for key 'PRIMARY' 2026-01-27 14:30:14.439 INFO 3374086 --- [ocp-schedules-24,db0777b817e5708d,8fd71e7171f14641] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2026-01-27 14:30:14.444 INFO 3374086 --- [metric-archive-async-46,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1769495340, nodeCount=9, affectRows=9 2026-01-27 14:30:14.448 WARN 3374086 --- [scanner-1878667-1,db0777b817e5708d,35bb0ce99831f1c6] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 15 2026-01-27 14:30:14.468 INFO 3374086 --- [metric-archive-async-34,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1769495340, nodeCount=23, affectRows=23 2026-01-27 14:30:16.245 WARN 3374086 --- [ocp-async-7,d6e9a8ca0f00ce8a,057e594f1c732abd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:30:16.245 INFO 3374086 --- [ocp-async-9,26bfe67804d13d64,0ffef616d306db71] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:30:16.308 INFO 3374086 --- [prometheus-minute-schedule-1,55937fbe6cc4e9c5,ae42344effc6abc8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495460, interval=60 2026-01-27 14:30:16.311 INFO 3374086 --- [prometheus-minute-schedule-1,55937fbe6cc4e9c5,ae42344effc6abc8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:30:16.540 INFO 3374086 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Follower checked 2026-01-27 14:30:17.041 INFO 3374086 --- [ocp-async-5,8d0a9f4e0b41a8c1,c779b66e2215afa3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2026-01-27 14:30:17.339 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:30:17.917 INFO 3374086 --- [ocp-schedules-10,27f6c7b02e716b62,7c6865ebc6ed7371] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2026-01-27 14:30:17.927 INFO 3374086 --- [ocp-schedules-10,27f6c7b02e716b62,7c6865ebc6ed7371] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2026-01-27 14:30:17.936 INFO 3374086 --- [ocp-schedules-10,27f6c7b02e716b62,7c6865ebc6ed7371] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2026-01-27 14:30:17.962 INFO 3374086 --- [ocp-schedules-10,27f6c7b02e716b62,7c6865ebc6ed7371] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2026-01-27 14:30:17.971 INFO 3374086 --- [ocp-schedules-10,27f6c7b02e716b62,7c6865ebc6ed7371] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2026-01-27 14:30:17.996 INFO 3374086 --- [ocp-schedules-10,27f6c7b02e716b62,7c6865ebc6ed7371] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2026-01-27 14:30:18.004 INFO 3374086 --- [ocp-schedules-10,27f6c7b02e716b62,7c6865ebc6ed7371] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2026-01-27 14:30:18.029 INFO 3374086 --- [ocp-schedules-10,27f6c7b02e716b62,7c6865ebc6ed7371] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=3, cachedSize=3 2026-01-27 14:30:18.246 INFO 3374086 --- [ocp-async-2,93d4a65dff9805db,fead0097bd0490cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:30:18.246 WARN 3374086 --- [ocp-async-4,ff1c5575f9f120bc,acb078c0bb18252f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:30:19.308 INFO 3374086 --- [prometheus-minute-schedule-1,26b97fba01f2f023,6495fd26ffa7358d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495460, interval=60 2026-01-27 14:30:19.311 INFO 3374086 --- [prometheus-minute-schedule-1,26b97fba01f2f023,6495fd26ffa7358d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:30:20.246 WARN 3374086 --- [ocp-async-8,fe5bf9af49fb792b,b5576972e477af14] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:30:20.246 INFO 3374086 --- [ocp-async-6,d0291b9e8fd8ecd6,0f88ca62c41b3274] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:30:20.308 INFO 3374086 --- [ocp-schedules-10,4852f7bcbdccbb03,c27a5b67af617090] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2026-01-27 14:30:20.339 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:30:22.246 INFO 3374086 --- [ocp-async-6,5e71fe21156090ea,18593ffde4b8c4e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:30:22.246 WARN 3374086 --- [ocp-async-5,9f92ce5660c75592,f1844b358dfc73ab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:30:22.308 INFO 3374086 --- [prometheus-minute-schedule-1,e754bd4d6197b0de,a8101d0561017492] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495460, interval=60 2026-01-27 14:30:22.311 INFO 3374086 --- [prometheus-minute-schedule-1,e754bd4d6197b0de,a8101d0561017492] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:30:23.340 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:30:24.246 INFO 3374086 --- [ocp-async-10,396bdc73249d2755,0e1156a623b1f847] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:30:24.246 WARN 3374086 --- [ocp-async-6,bed94bd409ef4b65,42411b749819e194] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:30:24.291 INFO 3374086 --- [ocp-schedules-15,0ed30950f052998f,bae979e99f36792e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2026-01-27 14:30:24.470 INFO 3374086 --- [ocp-schedules-29,073016a97d44d603,488f2f85dbcae8d5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2026-01-27 14:30:24.480 WARN 3374086 --- [scanner-1878799-1,073016a97d44d603,9ca4101124703425] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 15 2026-01-27 14:30:24.489 INFO 3374086 --- [alarm-detect-19,073016a97d44d603,447621daccee6853] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 513, data is empty. 2026-01-27 14:30:24.492 INFO 3374086 --- [alarm-detect-17,073016a97d44d603,f591e15158bad2b4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 513, data is empty. 2026-01-27 14:30:24.493 INFO 3374086 --- [alarm-detect-11,073016a97d44d603,dd7155ce8349906c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 513, data is empty. 2026-01-27 14:30:24.510 INFO 3374086 --- [alarm-detect-9,073016a97d44d603,990e80f746f66183] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 513, data is empty. 2026-01-27 14:30:25.308 INFO 3374086 --- [prometheus-minute-schedule-1,7aca68356e56759b,bb5707b1eb252e31] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495460, interval=60 2026-01-27 14:30:25.311 INFO 3374086 --- [prometheus-minute-schedule-1,7aca68356e56759b,bb5707b1eb252e31] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:30:26.121 INFO 3374086 --- [ocp-schedules-4,45ab944137639c1b,1fcbd409a39c53fc] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2026-01-27 14:30:26.122 INFO 3374086 --- [ocp-schedules-5,4246ae39b44f8f47,72c5245e05dd2b96] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2026-01-27 14:30:26.124 INFO 3374086 --- [ocp-schedules-24,298658dfba457b41,a2de7b1962b9c650] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2026-01-27 14:30:26.128 INFO 3374086 --- [ocp-schedules-16,93a558d3deeeb719,5f73cd8bf40612f7] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2026-01-27 14:30:26.134 INFO 3374086 --- [ocp-schedules-30,387360870326f5a8,4d17145e6754efd5] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.132.0.48] 2026-01-27 14:30:26.134 INFO 3374086 --- [ocp-schedules-2,fa44bcf5a75c8efc,3dc0c9191dc467c0] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.132.0.48] 2026-01-27 14:30:26.134 INFO 3374086 --- [ocp-schedules-18,c4ac5724e61c776e,6177b3221a9a3ac0] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.132.0.48] 2026-01-27 14:30:26.135 INFO 3374086 --- [host-check-clockdiff1,387360870326f5a8,f44ac031fa8beb1f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.132.0.48], timeoutMillis=20000 2026-01-27 14:30:26.135 INFO 3374086 --- [host-check-host-status1,fa44bcf5a75c8efc,189a24951c2d256a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.132.0.48, originalStatus=ONLINE 2026-01-27 14:30:26.135 INFO 3374086 --- [host-check-agent-status1,c4ac5724e61c776e,9632a82df8b21c3f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.132.0.48, originalStatus=ONLINE 2026-01-27 14:30:26.142 INFO 3374086 --- [host-check-host-status1,fa44bcf5a75c8efc,189a24951c2d256a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2026-01-27 14:30:26.143 INFO 3374086 --- [host-check-agent-status1,c4ac5724e61c776e,9632a82df8b21c3f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2026-01-27 14:30:26.151 INFO 3374086 --- [host-check-host-status1,fa44bcf5a75c8efc,189a24951c2d256a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2026-01-27 14:30:26.151 INFO 3374086 --- [host-check-agent-status1,c4ac5724e61c776e,9632a82df8b21c3f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2026-01-27 14:30:26.176 INFO 3374086 --- [host-check-clockdiff1,387360870326f5a8,f44ac031fa8beb1f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1769495426 -1 -1 , error= 2026-01-27 14:30:26.177 INFO 3374086 --- [host-check-host-status1,fa44bcf5a75c8efc,189a24951c2d256a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.132.0.48:62888/api/v1/time, params:null 2026-01-27 14:30:26.177 INFO 3374086 --- [host-check-agent-status1,c4ac5724e61c776e,9632a82df8b21c3f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.132.0.48:62888/api/v1/agent/status, params:null 2026-01-27 14:30:26.180 INFO 3374086 --- [host-check-host-status1,fa44bcf5a75c8efc,189a24951c2d256a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.132.0.48, timestamp=2026-01-27T14:30:26.178981938+08:00 2026-01-27 14:30:26.180 INFO 3374086 --- [host-check-host-status1,fa44bcf5a75c8efc,189a24951c2d256a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2026-01-27T14:30:26.180124507+08:00 2026-01-27 14:30:26.185 INFO 3374086 --- [host-check-agent-status1,c4ac5724e61c776e,9632a82df8b21c3f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.132.0.48 2026-01-27 14:30:26.185 INFO 3374086 --- [host-check-agent-status1,c4ac5724e61c776e,9632a82df8b21c3f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.132.0.48 2026-01-27 14:30:26.185 INFO 3374086 --- [host-check-agent-status1,c4ac5724e61c776e,9632a82df8b21c3f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.132.0.48 2026-01-27 14:30:26.186 INFO 3374086 --- [host-check-host-status1,fa44bcf5a75c8efc,189a24951c2d256a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2026-01-27T14:30:26.180124507+08:00 and status AVAILABLE 2026-01-27 14:30:26.188 INFO 3374086 --- [host-check-host-status1,fa44bcf5a75c8efc,189a24951c2d256a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2026-01-27 14:30:26.204 INFO 3374086 --- [host-check-host-status1,fa44bcf5a75c8efc,189a24951c2d256a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2026-01-27 14:30:26.212 INFO 3374086 --- [host-check-host-status1,fa44bcf5a75c8efc,189a24951c2d256a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2026-01-27 14:30:26.238 INFO 3374086 --- [host-check-host-status1,fa44bcf5a75c8efc,189a24951c2d256a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.132.0.48:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2026-01-27 14:30:26.246 WARN 3374086 --- [ocp-async-2,b3f8d72cd613d93a,1c27c6dcb2a78114] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:30:26.246 INFO 3374086 --- [ocp-async-8,dd1540d151703a1d,74ec71b32f012afd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:30:26.288 INFO 3374086 --- [host-check-host-status1,fa44bcf5a75c8efc,189a24951c2d256a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2026-01-27 14:30:26.307 INFO 3374086 --- [host-check-host-status1,fa44bcf5a75c8efc,189a24951c2d256a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2026-01-27 14:30:26.324 INFO 3374086 --- [host-check-host-status1,fa44bcf5a75c8efc,189a24951c2d256a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.132.0.48:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2026-01-27 14:30:26.340 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:30:26.365 INFO 3374086 --- [host-check-host-status1,fa44bcf5a75c8efc,189a24951c2d256a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2026-01-27 14:30:26.383 INFO 3374086 --- [host-check-host-status1,fa44bcf5a75c8efc,189a24951c2d256a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2026-01-27 14:30:26.400 INFO 3374086 --- [host-check-host-status1,fa44bcf5a75c8efc,189a24951c2d256a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.132.0.48:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2026-01-27 14:30:26.440 INFO 3374086 --- [host-check-host-status1,fa44bcf5a75c8efc,189a24951c2d256a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2026-01-27 14:30:26.458 INFO 3374086 --- [host-check-host-status1,fa44bcf5a75c8efc,189a24951c2d256a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2026-01-27 14:30:27.041 INFO 3374086 --- [ocp-async-10,e0d02c290ddb37fe,685d249be0f024ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2026-01-27 14:30:28.246 WARN 3374086 --- [ocp-async-5,6dc1b73383a87e15,8dacaeeac2442cf2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:30:28.246 INFO 3374086 --- [ocp-async-9,fd3432c28687a1c8,eee07306a5ed3a6f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:30:28.308 INFO 3374086 --- [prometheus-minute-schedule-1,8e916dc601d437b4,acc61b444e72580d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495460, interval=60 2026-01-27 14:30:28.311 INFO 3374086 --- [prometheus-minute-schedule-1,8e916dc601d437b4,acc61b444e72580d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:30:29.340 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:30:29.363 INFO 3374086 --- [ocp-schedules-24,6ec89d48b35c76e7,4dd6db6a780a3d0f] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2026-01-27 14:30:29.363 INFO 3374086 --- [ocp-schedules-24,6ec89d48b35c76e7,4dd6db6a780a3d0f] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2026-01-27 14:30:29.377 INFO 3374086 --- [Collect all tenants major compactions1,6ec89d48b35c76e7,9b30ab1961cc1bf1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myocp, tenant:ocp_monitor 2026-01-27 14:30:29.392 INFO 3374086 --- [Collect all tenants major compactions1,6ec89d48b35c76e7,9b30ab1961cc1bf1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2026-01-27 14:30:29.393 INFO 3374086 --- [Collect all tenants major compactions1,6ec89d48b35c76e7,9b30ab1961cc1bf1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2026-01-27 14:30:29.395 INFO 3374086 --- [Collect all tenants major compactions1,6ec89d48b35c76e7,9b30ab1961cc1bf1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2026-01-27 14:30:29.398 INFO 3374086 --- [Collect all tenants major compactions1,6ec89d48b35c76e7,9b30ab1961cc1bf1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2026-01-27 14:30:29.400 INFO 3374086 --- [Collect all tenants major compactions1,6ec89d48b35c76e7,9b30ab1961cc1bf1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myocp, tenant:ocp_monitor 2026-01-27 14:30:29.401 INFO 3374086 --- [ocp-schedules-24,6ec89d48b35c76e7,4dd6db6a780a3d0f] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2026-01-27 14:30:30.246 WARN 3374086 --- [ocp-async-9,609ca9aeb173d323,135b02d932a5eb9d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:30:30.247 INFO 3374086 --- [ocp-async-4,39d60c3fe1fdcf55,7ad15e5273180cb5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:30:30.263 INFO 3374086 --- [ocp-schedules-29,120f6aa6bf9cb34b,b7a0c1fee95352bd] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2026-01-27 14:30:30.310 INFO 3374086 --- [ocp-schedules-28,3d11eb9106e287b7,2b4d1de59a994c57] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2026-01-27 14:30:31.308 INFO 3374086 --- [prometheus-minute-schedule-1,48930bf03d5cfe66,f58587fe9fccc0a4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495460, interval=60 2026-01-27 14:30:31.311 INFO 3374086 --- [prometheus-minute-schedule-1,48930bf03d5cfe66,f58587fe9fccc0a4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:30:32.247 WARN 3374086 --- [ocp-async-2,6d006c5b6b939478,f40938cd36c489dc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:30:32.247 INFO 3374086 --- [ocp-async-1,1da4428948c8ecd6,3f2b9bd8431ad334] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:30:32.341 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:30:34.247 WARN 3374086 --- [ocp-async-8,940f001ec0bdf294,2f154e454e23af64] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:30:34.247 INFO 3374086 --- [ocp-async-7,18d6db78f7069972,36dfdc90b3c976d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:30:34.308 INFO 3374086 --- [prometheus-minute-schedule-1,7096fb37591ca995,b16dbeee1a812e76] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495460, interval=60 2026-01-27 14:30:34.311 INFO 3374086 --- [prometheus-minute-schedule-1,7096fb37591ca995,b16dbeee1a812e76] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:30:34.513 INFO 3374086 --- [ocp-schedules-4,b31601b06d4c57c5,858656d27fd3f271] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2026-01-27 14:30:34.521 WARN 3374086 --- [scanner-1878969-1,b31601b06d4c57c5,8f70dfdfff43ca93] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 15 2026-01-27 14:30:35.341 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:30:35.754 INFO 3374086 --- [ocp-schedules-13,fa5076eede911cb2,f8783fbd077f1764] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2026-01-27 14:30:35.754 INFO 3374086 --- [ocp-schedules-13,fa5076eede911cb2,f8783fbd077f1764] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2026-01-27 14:30:35.788 INFO 3374086 --- [ocp-schedules-13,fa5076eede911cb2,f8783fbd077f1764] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2026-01-27 14:30:35.791 INFO 3374086 --- [ocp-schedules-13,fa5076eede911cb2,f8783fbd077f1764] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2026-01-27 14:30:36.247 WARN 3374086 --- [ocp-async-3,3bd2cf2e0f8cc179,14f74f4659e6c688] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:30:36.247 INFO 3374086 --- [ocp-async-6,1c0d0a6cc003cfea,70f9e9480f467241] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:30:37.042 INFO 3374086 --- [ocp-async-9,40863991ce6c83b0,fd1748cb193069cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2026-01-27 14:30:37.308 INFO 3374086 --- [prometheus-minute-schedule-1,0d900e477a0f8247,4535841727fb6a48] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495460, interval=60 2026-01-27 14:30:37.311 INFO 3374086 --- [prometheus-minute-schedule-1,0d900e477a0f8247,4535841727fb6a48] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:30:38.247 WARN 3374086 --- [ocp-async-1,98afa4afd7abbdae,b54a74a1f6cecb82] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:30:38.247 INFO 3374086 --- [ocp-async-2,a1e3e9a31ab81099,64bf5ac8f889fd78] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:30:38.342 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:30:40.002 INFO 3374086 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2026-01-27 14:30:40.002 INFO 3374086 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2026-01-27 14:30:40.247 WARN 3374086 --- [ocp-async-5,bda1b155a59add42,ac353bfb95412469] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:30:40.247 INFO 3374086 --- [ocp-async-7,782bd4b9a9acad79,c0f15ce337ad5412] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:30:40.308 INFO 3374086 --- [prometheus-minute-schedule-1,6405bedb8027f6bc,bda284626125a08e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495460, interval=60 2026-01-27 14:30:40.310 INFO 3374086 --- [prometheus-minute-schedule-1,6405bedb8027f6bc,bda284626125a08e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:30:40.312 INFO 3374086 --- [ocp-schedules-12,5f351bba20854668,5ee638e6717ca3c7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2026-01-27 14:30:41.342 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:30:42.247 WARN 3374086 --- [ocp-async-3,9d00a7e82ffaeac3,b711a32bc3fa14c1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:30:42.247 INFO 3374086 --- [ocp-async-9,9b291c5e7e5f2046,98d3c7393eddfb3c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:30:43.308 INFO 3374086 --- [prometheus-minute-schedule-1,158fda0f377a6336,a4e6392daaaad1d1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495460, interval=60 2026-01-27 14:30:43.311 INFO 3374086 --- [prometheus-minute-schedule-1,158fda0f377a6336,a4e6392daaaad1d1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:30:44.247 INFO 3374086 --- [ocp-async-1,59d5792f2ca0f857,08ed2dc882c12b6b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:30:44.247 WARN 3374086 --- [ocp-async-10,e5da89bfd221b5aa,e511c3aeb10b2942] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:30:44.294 INFO 3374086 --- [ocp-schedules-19,eece379b5b2f554a,a3bce35a49b80e22] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2026-01-27 14:30:44.343 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:30:44.544 INFO 3374086 --- [ocp-schedules-10,6deb8d2abb3e74bd,a932f863020d4375] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2026-01-27 14:30:44.551 WARN 3374086 --- [scanner-1879091-1,6deb8d2abb3e74bd,d7181ccce97e59fc] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 15 2026-01-27 14:30:44.562 INFO 3374086 --- [alarm-detect-6,6deb8d2abb3e74bd,151745377d6e8a60] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 513, data is empty. 2026-01-27 14:30:44.563 INFO 3374086 --- [alarm-detect-7,6deb8d2abb3e74bd,3230d1e68f9b8787] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 513, data is empty. 2026-01-27 14:30:44.564 INFO 3374086 --- [alarm-detect-13,6deb8d2abb3e74bd,9472bd7e45a29c04] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 513, data is empty. 2026-01-27 14:30:44.582 INFO 3374086 --- [alarm-detect-8,6deb8d2abb3e74bd,d7f7db255d8f7ad6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 513, data is empty. 2026-01-27 14:30:46.248 INFO 3374086 --- [ocp-async-5,6c9ad5e5506e1a8b,5504519be212f0b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:30:46.248 WARN 3374086 --- [ocp-async-8,d1693a2a15d70b11,5a17015ec6d9eb74] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:30:46.308 INFO 3374086 --- [prometheus-minute-schedule-1,cb8058c711b7db75,14a4e620369ba970] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495460, interval=60 2026-01-27 14:30:46.311 INFO 3374086 --- [prometheus-minute-schedule-1,cb8058c711b7db75,14a4e620369ba970] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:30:47.042 INFO 3374086 --- [ocp-async-2,6c10c605e1537c50,1349a0fb7cc94fb3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2026-01-27 14:30:47.186 INFO 3374086 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:PLAN_CHANGE_DETECTOR, skipped 2026-01-27 14:30:47.191 INFO 3374086 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:SQL_INDEX_DIAGNOSER, skipped 2026-01-27 14:30:47.229 INFO 3374086 --- [pool-20-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:SQL_PERF_DIAGNOSER, skipped 2026-01-27 14:30:47.310 INFO 3374086 --- [prometheus-cache-monitor1,cb112568a530f927,28245ae9caf2a5ae] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":3100,"minuteWriteQueue":0,"minuteReadCacheSize":1173,"secondWriteCacheSize":774,"seriesIdKeySize":5585,"secondWriteQueue":0,"minuteWriteCacheSize":1177} 2026-01-27 14:30:47.310 INFO 3374086 --- [prometheus-cache-monitor1,cb112568a530f927,28245ae9caf2a5ae] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_query_rt_total_cumulative_count=29, ob_sysstat_40012=17, ob_server_resource_memory_assigned_bytes=10, node_memory_sreclaimable_bytes=12, ob_sysstat_40011=17, ob_sysstat_40010=17, plan_max_cache_item_size=9, ob_unit_num=10, tcp_retrans=13, ob_database_disk_required_size=32, go_memstats_mspan_sys_bytes=10, go_memstats_stack_sys_bytes=10, exporter_export_metrics_bytes_total=17, node_filesystem_size_bytes=45, http_server_requests_seconds_count=91, ob_tenant_request_queue_total_size=16, go_gc_gogc_percent=10, jvm_gc_memory_allocated_bytes_total=4, ob_compaction_frozen_time=12, ob_system_event_total_waits=26, ob_tenant_disk_v4_log_disk_size=12, ob_sysstat_50050=17, log_stream_degraded_count=16, ob_sysstat_40019=17, ob_sysstat_40018=17, processor_process_metrics_total=14, ob_plan_cache_hit_total=16, jvm_gc_max_data_size_bytes=4, node_filesystem_readonly=44, go_memstats_frees_total=10, io_await=35, host_available=7, ob_sql_event_in_sequence_load=16, go_memstats_mallocs_total=10, ob_unit_config_max_memory_bytes=19, ob_sysstat_200006=17, ob_database_disk_data_size=31, io_util=35, ob_sysstat_200005=17, ob_sysstat_200002=17, primary_leaders_cnt=15, ob_sysstat_200001=17, ob_tenant_memstore_freeze_cnt=10, ob_tenant_disk_v4_data_size=12, ob_query_rt_table_total_cumulative_count=29, ob_tenant_unit_min_memory=26, ob_sysstat_60087=17, ob_memstore_total_bytes=20, jvm_threads_states_threads=10, process_resident_memory_bytes=10, ob_cache_size_bytes=30, ob_tenant_assigned_cpu_total=16, ob_tenant_unit_max_memory=26, input_collect_metrics_total=15, ob_sysstat_50039=17, ob_sysstat_60083=17, ob_sysstat_50038=17, node_boot_time_seconds=12, cpu_count=13, node_disk_io_time_weighted_seconds_total=18, node_disk_write_time_seconds_total=18, output_write_metrics_total=11, ob_sysstat_50035=17, ob_cluster_sync=7, ob_sysstat_50034=17, process_open_fds=10, ob_sysstat_50037=17, ob_sysstat_50036=17, process_memory_rss_bytes=17, http_status_count=14, process_max_fds=10, ob_server_resource_cpu_assigned_percent=10, ob_sysstat_50030= 2026-01-27 14:30:47.343 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:30:48.248 INFO 3374086 --- [ocp-async-3,adff5f58f065a649,ec9b79bd4ef01c41] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:30:48.248 WARN 3374086 --- [ocp-async-9,5765857c17188a12,f35d08d494013ca6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:30:48.347 INFO 3374086 --- [ocp-schedules-21,af134c3669d4ab52,bc50c1c0df364e4c] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.132.0.48] 2026-01-27 14:30:48.347 INFO 3374086 --- [ocp-schedules-21,af134c3669d4ab52,bc50c1c0df364e4c] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2026-01-27 14:30:48.354 INFO 3374086 --- [ocp-schedules-21,af134c3669d4ab52,bc50c1c0df364e4c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2026-01-27 14:30:48.362 INFO 3374086 --- [ocp-schedules-21,af134c3669d4ab52,bc50c1c0df364e4c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2026-01-27 14:30:48.389 INFO 3374086 --- [ocp-schedules-21,af134c3669d4ab52,bc50c1c0df364e4c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.132.0.48:62889/api/v1/log/alarms, params:null 2026-01-27 14:30:48.391 INFO 3374086 --- [ocp-schedules-21,af134c3669d4ab52,bc50c1c0df364e4c] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2026-01-27 14:30:49.308 INFO 3374086 --- [prometheus-minute-schedule-1,fda759faf7047168,0a21cac7c6100a50] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495460, interval=60 2026-01-27 14:30:49.311 INFO 3374086 --- [prometheus-minute-schedule-1,fda759faf7047168,0a21cac7c6100a50] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:30:50.248 WARN 3374086 --- [ocp-async-10,3bde7d72a681472b,8e0a3c64e687cb08] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:30:50.248 INFO 3374086 --- [ocp-async-4,4bc5d8f91df6272c,630b3868e393b641] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:30:50.315 INFO 3374086 --- [ocp-schedules-8,a806d63426afc761,1008e1e6fbbb72d3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2026-01-27 14:30:50.343 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189 2026-01-27 14:30:51.871 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2026-01-27 14:30:51.871 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] c.o.ocp.core.property.PropertyService : Refresh config properties 2026-01-27 14:30:51.876 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] c.o.o.b.spring.BootstrapRunListener : new: [] 2026-01-27 14:30:51.876 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] c.o.o.b.spring.BootstrapRunListener : init spring 2026-01-27 14:30:51.876 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2026-01-27 14:30:51.876 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] c.o.o.b.spring.BootstrapRunListener : starting 2026-01-27 14:30:51.880 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] c.o.o.b.spring.BootstrapRunListener : new: [] 2026-01-27 14:30:51.880 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] c.o.o.b.spring.BootstrapRunListener : init spring 2026-01-27 14:30:51.880 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2026-01-27 14:30:51.880 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] c.o.o.b.spring.BootstrapRunListener : starting 2026-01-27 14:30:52.026 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2026-01-27 14:30:52.029 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2026-01-27 14:30:52.029 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2026-01-27 14:30:52.029 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2026-01-27 14:30:52.029 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.4.0 2026-01-27 14:30:52.029 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.4.0 2026-01-27 14:30:52.034 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2026-01-27 14:30:52.220 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2026-01-27 14:30:52.248 WARN 3374086 --- [ocp-async-8,ee656c34d8fb9929,d9268c63881e8050] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2026-01-27 14:30:52.248 INFO 3374086 --- [ocp-async-2,e81d93d9aacfc1d8,fd01256fc412a41f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2026-01-27 14:30:52.267 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.387S 2026-01-27 14:30:52.267 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2026-01-27 14:30:52.308 INFO 3374086 --- [prometheus-minute-schedule-1,e76bea9fdebca527,d97fc8d470c1df46] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1769495460, interval=60 2026-01-27 14:30:52.311 INFO 3374086 --- [prometheus-minute-schedule-1,e76bea9fdebca527,d97fc8d470c1df46] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=2, collectTaskCount=2 2026-01-27 14:30:52.429 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2026-01-27 14:30:52.429 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-ocp-default'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}] 2026-01-27 14:30:52.464 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2026-01-27 14:30:52.464 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2026-01-27 14:30:52.464 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2026-01-27 14:30:52.464 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.4.0 2026-01-27 14:30:52.464 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.4.0 2026-01-27 14:30:52.465 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2026-01-27 14:30:52.470 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2026-01-27 14:30:52.473 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] o.s.boot.SpringApplication : Started application in 0.597 seconds (process running for 57222.105) 2026-01-27 14:30:52.473 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.597S 2026-01-27 14:30:52.971 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] o.s.cloud.endpoint.RefreshEndpoint : Refreshed keys : [ocp.internal.rpc.user.hash, ocp.internal.rpc.user.password] 2026-01-27 14:30:52.989 WARN 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm type system_parameter_not_taken_effect is not enabled 2026-01-27 14:30:52.989 INFO 3374086 --- [ocp-schedules-2,2a3bbab9ac1276d7,a704e5a40d8481ab] c.o.ocp.core.property.PropertyService : Refresh config properties done 2026-01-27 14:30:53.344 INFO 3374086 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.132.0.49:8189